Incident Summary Report

Issue: API Performance Degradation
Date: 2021-03-08

Description

On the morning of March 8, 2021, WorkOS received a rapid burst of API usage, several orders of magnitude greater than the typical usage of WorkOS APIs. This caused a performance issue with API load times that resulted in authentication timing-out for users. The impact occurred from 10:01am EST until 10:21 EST. Our investigation shows that during this time window, users were still able to successfully log in, but many users experienced increased latency and a degradation of service.

Security Impact

There was no security impact during this incident.

Immediate Actions Taken

• Once the identified issue was discovered a code fix was released to address the root cause of the performance degradation.
• Actively monitored the API post release to confirm the underlying issue was addressed.

WorkOS Mitigation Strategy

• New policies will be put in place to ensure timely and proactive outbound customer communication.
• Investigate ways on how to handle error cases to improve the customer experience

• Increase diversity of load testing to include datasets and situations involving continually increasing load and data growth

  • Including auditing SSO critical path.