From 2023-08-28 23:22 UTC to 2023-08-29 02:22 UTC WorkOS’s SSO product was unavailable to users through custom domains. Requests returned HTTP 403 Forbidden errors.

We understand that WorkOS sits on a critical path for our customers’ applications. This is not a responsibility we take lightly and this outage is not in line with the service we aim to provide. We are taking all necessary steps to ensure an incident like this does not happen again.

Who was affected?

The incident affected SSO API requests for users with custom hostnames. Affected requests during this time resulted in 403 errors and displayed the error message “This web property is not accessible via this address.”

What happened?

While performing maintenance on our Web Application Firewall, a new set of rules were applied to production. This change marked some legitimate requests as anomalous. Alerts were not properly configured to notify engineers when seeing a spike in anomalous 4XX traffic.

The main factor that led to this incident was improper controls around how production Web Application Firewall changes should be applied.

What will we do to mitigate problems like this in the future?

Moving forward, WorkOS will take the following actions:

1. Establish additional access control policies around applying Web Application Firewall changes.
2. Add monitoring around increases in anomalous traffic.
3. Add monitoring around failures with custom hostnames.