Preparing for the worst: Our core database failover test
By Steven Van ·
Vercel failed over its control-plane database from Azure West US to East US 2 on 24 July 2025 with zero customer downtime.
On 24 July 2025, Vercel ran a full production failover of its core control-plane database, moving the primary write region from Azure West US to East US 2. The test covered every API request, background job, deployment, and build operation on the platform, with no impact on customer-facing production traffic.
- Preview and development traffic routing was affected during the switch. Production CDN traffic, served by a separate globally-replicated DynamoDB architecture across Vercel's 19 regions, stayed isolated and unaffected.
- The failover started at 14:09 UTC and finished by 14:23; it was reversed an hour later. About 500 builds were disrupted, mostly Vercel's own, and one customer (Vercel itself) saw elevated latencies tied to a particularly large database partition.
- About 2% of write traffic never fully completed the transition to the new region, and some clients saw the write region flip back and forth during the switch.
The exercise followed 57 rehearsals in staging, where Vercel found issues like a hardcoded five-minute delay before some internal services recognized a new database write region; that delay was patched down to five milliseconds. This was a graceful, reversible failover. Vercel noted that a real incident would instead trigger a more aggressive "Offline Region" operation, which can't be reversed for up to two business days and which it has rehearsed twice in staging.
