How AI Gateway runs on Fluid compute
By Steven Van ·
Vercel details how Active CPU Pricing and Fluid's instance reuse let AI Gateway pay CPU rates for under 8% of its runtime.
AI Gateway, the service that lets developers connect to hundreds of AI models through one interface, runs on Vercel's Fluid compute. In its first month of general availability, the gateway logged roughly 16,000 total runtime hours, but only 1,200 of those hours involved actual CPU work like processing requests, routing logic, and streaming responses. The other 14,800 hours were spent waiting on responses from AI providers.
With Fluid and Active CPU Pricing, Vercel charges CPU rates only while the CPU is running and a lower memory-only rate the rest of the time, which for AI Gateway means paying CPU rates for less than 8% of runtime instead of 100%. Fluid also reuses instances while they're running, not just between invocations, so one request can pause to wait on a provider's response while another executes on the same instance, keeping frequently accessed provider routes, credentials, and quota snapshots in memory to cut down on Redis lookups. Each Vercel region keeps its own Redis cluster for quota tracking and usage updates, and every Fluid instance feeds health checks and in-memory statistics back to a monitoring system that can reroute traffic if a provider region starts responding slowly.
