Client RPC Latency in Exchange 2010

This morning we had an issue where clients using one exchange server were experiencing tremendous latency when using Outlook, OWA or ActiveSync.  The RPC Latency should never exceed 250ms, and this server was running around 2000ms.  After some troubleshooting, I was able to clearly identify the cause of the problem for the Exchange 2010 server….  Turns out that the iSCSI interface was taking about 11 times as long as the other Exchange servers to process average requests.  The morning started with only a 200ms response time and then grew quickly as the day went on.  The other exchange servers were taking 0.115 (115ms) for the average request processing time.  This server was running 1.305 on average and had delays of up to 7.6 seconds!!  Databases do not like waiting 7.6 seconds for the disk to respond.

Turns out that the NetApp provided software had overridden the MS iSCSI mounting.  As a result, the iSCSI traffic was being sent directly out of the main network card used for all MAPI (Outlook) traffic.  That created a downward spiral with each traffic type slowing because it was fighting the other.

Since this is a 6 node DAG, we switched-over the users to another node.  Middle of the day, executives and all, no downtime.  Interestingly, the server that was now handling double load with around 2500 users was running with a 7-20ms response time for RPC traffic.

To me it proves just how powerful Exchange is at handling IO traffic.  When transferred to a healthy server, several thousand users with around 2.5TB of mail are running without any issues on iSCSI.

This entry was posted in Exchange 2010. Bookmark the permalink. Both comments and trackbacks are currently closed.