We're sorry for the disruption this incident caused for you and your users. Authentication is a critical path, and during this window we did not meet the reliability standard you expect from WorkOS. We take responsibility for the impact. Service was restored the same day, and we have already shipped several lasting fixes while continuing broader cleanup and detection work.

Postmortem: AuthKit custom domain sign-in failures

Date: September 9, 2026

Duration: 21:09 – 22:30 UTC (~80 minutes of customer impact)

Status: Resolved

Summary

On September 9, 2026, from approximately 21:09 to 22:30 UTC, a small subset of customers using AuthKit custom domains experienced failures when submitting authentication forms (sign-in, sign-up, MFA verification, device authorization, and email verification). Users on affected domains saw HTTP 500 errors on those form submissions, which blocked interactive login for the duration of the incident.

This was a partial outage. It affected a small cohort of AuthKit custom domains provisioned through an earlier self-serve configuration path—about 0.17% of AuthKit custom-domain traffic was actually affected by the failure. Default AuthKit domains (*.authkit.app), the WorkOS API, the Dashboard, and GET requests (page loads and most OAuth callback traffic) on custom domains continued to work. The failure mode was specific to server-action POST requests used by AuthKit hosted UI forms.

We first learned of the issue from a customer report at approximately 22:04 UTC (~55 minutes after impact began). Engineering restored service by 22:30 UTC. No evidence of unauthorized access, data exposure, or data corruption was found.

Background

AuthKit can be hosted on a customer’s own hostname (a custom domain). Traffic for those hostnames is terminated at our edge and forwarded to the AuthKit application with HTTP headers that identify the original host and browser origin. The AuthKit application relies on those headers so that browser security checks for interactive form submissions (Next.js server actions) succeed.

Over time, WorkOS evolved how custom-domain traffic is routed at the edge—from earlier per-domain configuration toward shared, list-based edge rules and a newer application hosting path. During that migration, a set of legacy per-domain header rules remained in place for domains created in an earlier era. A later zone-wide edge rule normally overwrote those headers so the application always saw a consistent, CSRF-safe origin. While that zone-wide rule was enabled, the older per-domain rules appeared inert.

What happened

As part of decommissioning older edge configuration no longer needed for the current hosting path, the zone-wide header transform rule was disabled in production. That change unmasked the leftover per-domain rules for the older custom-domain cohort.

For those domains, the edge continued to rewrite the browser Origin header to the tenant’s default AuthKit hostname, while the application still received the customer’s custom hostname as the forwarded host. AuthKit’s application layer correctly rejected the mismatched server-action POSTs as invalid (CSRF / same-origin protection), returning HTTP 500 before application business logic ran.

Scope

• Structurally exposed: AuthKit custom domains still carrying the legacy per-domain header configuration (on the order of a few dozen hostnames).
• Observed traffic impact: a subset of those domains that received form POST traffic during the window (about sixteen hostnames with material volume).
• Scale: roughly 2% of AuthKit custom-domain traffic comes from potentially affected domains; ~3,100 failed server-action POSTs over the ~80-minute window. For affected domains, failure rates on form POSTs were often near 100%.

Symptoms

• Users on affected custom domains could not complete interactive AuthKit forms (sign-in and related flows).
• Default AuthKit hostnames and non-form traffic paths remained healthy.

What went wrong

The proximate trigger was disabling a shared edge header rule that had been silently correcting residual per-domain header rewrites left over from an earlier custom-domain design. The durable underlying condition was incomplete cleanup of that legacy configuration after the supported path moved to list-based rules—and the absence of automated detection for this failure shape.

Why it wasn’t caught sooner

Several detection gaps stacked:

1. Pre-change validation and canaries primarily covered newer custom domains (and often GET probes). This failure only appears on server-action POSTs for the legacy cohort, with no representation of this configuration in our validation environment.
2. Aggregate AuthKit error rates stayed relatively low because only a small share of traffic was affected, so fleet-wide thresholds did not page.

As a result, detection depended on customer reports rather than an automated page.

Remediation

Completed since the incident

• Worker safety net — Added application-edge protection that detects and drops stale per-domain origin values so legacy rules can be retired without recreating this mismatch, even if rule order differs.
• Staging validation path — Reproduced the legacy custom-domain configuration in staging so further edge cleanup can be proven safe before production changes.

In progress / planned

• Remove legacy per-domain edge rules for AuthKit custom domains after staged validation, so shared transform rules are no longer load-bearing masks.
• Inventory residual edge configuration tied to custom domains (including orphaned rules) so leftover resources have clear ownership and review.
• Per-customer / concentrated-failure detection so a small set of fully down tenants is visible even when fleet-wide error rates look healthy.
• Stronger change gates for catch-all / list edge-rule changes: same-phase inventory of earlier rules that can match the same hosts, and canaries that include legacy-era domains and server-action POST probes—not only GET checks on newer domains.
• Improved change visibility for production edge-rule state (today some zone rules are dashboard-managed without a PR trail).

We treat restoring the shared transform as necessary containment, not the durable end state. The combination of residue removal, better probes, and better detection is intended to prevent both recurrence of this exact mechanism and silent customer-total failures on a small hostname set.

Closing

A configuration cleanup on the AuthKit custom-domain edge path exposed leftover rules from an earlier design and blocked interactive sign-in for a limited set of custom domains for about eighty minutes. We restored service quickly once the issue was identified, and we are following through so legacy edge residue cannot remain load-bearing, so validation covers the domains and request types that actually break, and so concentrated authentication failures are visible to us before they are visible only to your users.

We regret the impact on affected customers and their end users, and we appreciate the reports that helped us engage. If you have questions about whether your integration was in the affected cohort, please contact WorkOS support (support@workos.com).