Some API endpoints were erroneously rate-limited

Incident Report for WorkOS

Postmortem

Summary

On Wednesday, July 23, between 17:50 and 22:21 UTC, several WorkOS API endpoints applied rate limits more aggressively than intended for some customers. The affected services were AuthKit, Directory Sync, Audit Logs, and the Events API.
Our published limits (see workos.com/docs/reference/rate-limits) remained unchanged, but a recent code change introduced a bug in the rate‑limiting service that prematurely returned HTTP 429 responses. Once identified, we rolled back the change and restored normal operation.
We understand the seriousness of this disruption and remain committed to delivering the highest level of reliability across our platform.

Root Cause Analysis

WorkOS enforces rate limits on API endpoints across several products (AuthKit, Audit Logs, Directory Sync, and the Events API) to ensure reliable and predictable uptime. We apply rate limiting at multiple layers of our infrastructure.
A recent code change introduced a bug in the application‑layer rate‑limiting service, which throttled certain traffic more aggressively than documented. The deployment process did not catch this bug because the enforcement logic executed before the logging component captured and recorded the request.

Remediation

Rate limits operate on the critical path of our services. In response to this incident, we are strengthening our testing, deployment, and observability safeguards across all public API endpoints. This includes capturing observability logs before any other endpoint code executes.

Posted Jul 23, 2025 - 19:45 EDT

Resolved

All services are operational and the incident has been resolved.
Posted Jul 23, 2025 - 19:17 EDT

Monitoring

We’ve rolled out the fix and are continuing to monitor for errors.
Posted Jul 23, 2025 - 19:10 EDT

Identified

Between 5:50 PM and 10:21 PM UTC, certain API endpoints across AuthKit, Audit Logs, Directory Sync, and the Events API were rate-limited more aggressively than intended.
We’ve identified the root cause, deployed a fix, and are currently monitoring the system.
Posted Jul 23, 2025 - 19:09 EDT
This incident affected: Core Services (Directory Sync, Audit Logs, AuthKit).