Earlier this week SurePrep customers experienced significantly degraded performance in the FileRoom application during peak hours. The cause of this issue was traced back to network communication problems between front-line web servers and session-management systems. Those communication problems were the result of routine Data Center updates to Citrix Xenserver, and therefore only affected virtual machines.
The virtual machine issue was not immediately apparent to SurePrep engineers, as all other systems were operating normally. Once the cause of the FileRoom performance degradation was determined, SurePrep technical staff migrated all functionality to “bare metal” hardware systems, which resolved the performance issues.
SurePrep is working closely with IBM to gain a full understanding of what went wrong with the virtual machine infrastructure and how such events can be prevented in the future. Additionally, SurePrep engineers are reviewing the existing Disaster Recovery plan to identify how to more quickly restore customer access in the event of an infrastructure failure. The plan currently relies on deploying systems to multiple Data Centers to ensure availability, but does not contemplate an infrastructure issue spanning all Data Centers.