SurePrep System Slowness

Incident Report for SurePrep

Postmortem

Earlier this week SurePrep customers experienced significantly degraded performance in the FileRoom application during peak hours. The cause of this issue was traced back to network communication problems between front-line web servers and session-management systems. Those communication problems were the result of routine Data Center updates to Citrix Xenserver, and therefore only affected virtual machines.

The virtual machine issue was not immediately apparent to SurePrep engineers, as all other systems were operating normally. Once the cause of the FileRoom performance degradation was determined, SurePrep technical staff migrated all functionality to “bare metal” hardware systems, which resolved the performance issues.

SurePrep is working closely with IBM to gain a full understanding of what went wrong with the virtual machine infrastructure and how such events can be prevented in the future. Additionally, SurePrep engineers are reviewing the existing Disaster Recovery plan to identify how to more quickly restore customer access in the event of an infrastructure failure. The plan currently relies on deploying systems to multiple Data Centers to ensure availability, but does not contemplate an infrastructure issue spanning all Data Centers.

Posted Oct 10, 2019 - 09:38 PDT

Resolved

We identified the root cause of the connectivity issues and deployed a fix. FileRoom connectivity has been restored. We continue to monitor the situation. Incident updates will be posted here as we get them.

Posted Oct 09, 2019 - 12:22 PDT

Update

Message from SurePrep President & CEO:

We are aware of the connectivity issues customers are experiencing when logging into the SurePrep FileRoom. Our teams are working tirelessly to resolve these technical issues as soon as possible. I’m confident we will diagnose and resolve the root cause of the concern and restore system performance soon. We are truly sorry for the disruption that you are experiencing. SurePrep’s consistent uptime and uninterrupted availability is pivotal to your processes, and we are working diligently to regain your trust. Periodic updates will be posted here on the SurePrep Status page as they become available.

Respectfully,

David Wyle
CEO/President

Posted Oct 09, 2019 - 09:30 PDT

Update

We continue to work towards a full resolution of the connection issues. As a short-term workaround, our customer support team can provide a PBFX file for binders you need to open. Once the PBFX file is downloaded, you can double click on it, enter your login credentials, and the binder will open. To request instructions for the short-term workaround please ask your staff to email support@sureprep.com with the subject “PBFX workaround” and include the Binder ID and/or Client ID. Support will reply with instructions.

Posted Oct 09, 2019 - 07:04 PDT

Update

We are seeing improvements in the connectivity issues and we are working around the clock towards a full resolution. Incident updates will be posted here as we get them.

Posted Oct 08, 2019 - 23:11 PDT

Update

We continue to work towards a full resolution with the highest priority. We sincerely apologize for the system slowness and outage. Incident updates will be posted here as we get them.

Posted Oct 08, 2019 - 16:43 PDT

Update

We continue to work towards a resolution of the connection issues. As a short-term workaround, our customer support team can provide a PBFX file for binders you need to open. Once the PBFX file is downloaded, you can double click on it, enter your login credentials, and the binder will open. To request instructions for the short-term workaround please ask your staff to email support@sureprep.com with the subject “PBFX workaround”. Support will reply with instructions.

Posted Oct 08, 2019 - 14:09 PDT

Update

Our engineers are making incremental FileRoom configuration changes and we are working with Microsoft support to fix the connection issues as soon as possible. We apologize for the inconvenience. Incident updates will be posted here as we get them.

Posted Oct 08, 2019 - 12:15 PDT

Update

Some customers are reporting system slowness and difficulty logging into FileRoom. Our operations team is investigating the cause. We apologize for the inconvenience. Incident updates will be posted here as we get them.

Posted Oct 08, 2019 - 07:43 PDT

Update

We are continuing to work on a fix for this issue.

Posted Oct 08, 2019 - 07:43 PDT

Update

We are continuing to work on a fix for this issue.

Posted Oct 08, 2019 - 07:42 PDT

Identified

We're seeing improvement in the system slowness that some customers have reported. Maintenance will be performed tonight to further mitigate the slowness. Incident updates will be posted here as we get them.

Posted Oct 07, 2019 - 17:57 PDT

Investigating

Some customers are reporting system slowness navigating FileRoom and opening SPbinder. Our operations team is investigating the cause. We apologize for the inconvenience. Incident updates will be posted here as we get them.

Posted Oct 07, 2019 - 12:55 PDT

This incident affected: FileRoom and SPbinder.