Incident Summary Report

Issue: Delayed Delivery of Queued Events
Date: 2021-03-08

Description

On the morning of March 8, 2021, WorkOS received a rapid burst of API usage, several orders of magnitude greater than the typical usage of WorkOS APIs. This resulting load caused an increase in downstream queue processing. The shape of this traffic was unique resulting in the identification of a hot path within background processing. During the timeframe of 10:01 AM EST until 1:02 PM EST downstream webhooks and magic links were delayed.

All messages were eventually delivered and no data was lost.

Security Impact

There was no security impact during this incident.

Immediate Actions Taken

• Once the identified issue was discovered a code fix was released to resolve a third party dependency that was bottlenecking the queue.
• Released code changes to handle how the third party calls were being initiated to remove the discovered hot path.

WorkOS Mitigation Strategy

• New policies will be put in place to ensure timely and proactive outbound customer communication.
• Identify failure cases for all third party outbound queue processing.