Sort providers by cost, latency, or throughput on AI Gateway
By Steven Van ·
A new `sort` option in AI Gateway ranks providers by cost, time to first token, or tokens per second, computed fresh at request time.
AI Gateway's default provider order blends reliability, output quality, cost, and response speed. A new sort option in Vercel's AI Gateway lets developers rank providers by a single metric instead, set as sort under providerOptions.gateway.
- cost: sorts by the provider's listed input price per million tokens, lowest first.
- ttft: sorts by median time to first token, lowest first.
- tps: sorts by median tokens per second throughput, highest first.
Providers are tried in the sorted order, with the gateway falling back to the next one if a provider fails. Sort can be combined with the existing order and only options, and ranking is computed at request time, so new providers, price changes, or latency shifts apply without code changes. Degraded providers get a penalty that pushes them down the list, and providers marked down are always sorted last regardless of their metric.