Google Workspace Users Incorrectly Deleted Incident Report

Between 20:08 UTC on October 19, 2021 and 17:14 UTC on October 20, 2021, a subset of Google Workspace Directories were impacted by an issue that incorrectly removed users from their Directories and triggered corresponding deletion and update webhook events. We apologize for the negative impact this had on our customers.

Who was affected?

This issue impacted Google Workspace Directories with more than 100 users. All other Directory types were not impacted by this issue.

What happened?

On Monday, October 18th, 2021 at 16:55 UTC we deployed a change to Directory Sync that introduced a bug in our job re-queuing logic. For Google Workspace directories with multiple pages of users (where a page is 100 users), syncing the directory would enqueue a new job for each page of users. When these jobs ran the next day, the presence of multiple, simultaneously-executing jobs created a race condition which caused some users in the Directory to be incorrectly deleted inside WorkOS.

We released a fix at 17:14 UTC on October 20th, 2021 that prevented these forked synchronization jobs from being scheduled and re-synchronized all Google Workspace directories to restore the integrity of our data.

What will we do to mitigate problems like this in the future?

We are taking steps to further ensure that multiple synchronization jobs cannot exist for a single Directory to prevent this race condition from reoccurring.

Investigation into this issue was complicated by updated ETag properties on users from Google registering as a change to the user’s data. This caused extraneous dsync.user.updated events to be issued by us. We have updated our change reconciliation to ignore this ETag.