How we rebuilt provider routing for better, faster inference
By Steven Van ·
The new default router weighs price, speed and reliability, cutting capacity errors 67% and lifting throughput 16.3% for a 6.0% cost increase.
OpenRouter has rebuilt the default provider routing behind OpenRouter. Until now, requests that did not set a sort or provider list went mostly to the provider with the lowest input token price. The new algorithm monitors every provider endpoint and balances cost, speed and reliability for each new request. The company describes it as the biggest change to its routing algorithm since it was founded, and says no code changes are needed. The details are in the announcement.
Why price-only routing was replaced
About 77% of requests on OpenRouter go to models served by more than one provider. For those, the old router ranked almost entirely on input token price, and failed requests were re-routed to the next cheapest option. OpenRouter lists these problems with that approach:
- Cheap providers won traffic even when they performed badly, and users waited through fallbacks.
- Rankings were static, so a provider had to degrade badly before traffic moved away.
- Providers could only win traffic by cutting price, which put pressure on their infrastructure spending.
- A provider could list a low input price and recover the difference through expensive cached or completion tokens.
How the Routing Cost works
The router calculates a real-time Routing Cost for each provider endpoint. It combines the price in dollars, the cost of waiting for a response, and the cost of retries when a provider fails or returns an unusable response.
- Effective price: based on recent measured traffic, including the provider's real cache hit rate and the model's average mix of input and output tokens, so it can differ a lot from list price.
- Throughput: measured as output tokens per second end to end. A faster provider can charge more and still win. Waiting time is converted to cost at a rate set per model.
- Time to first token: counted separately as well, because interactive use and agent sessions with many tool calls depend on it.
- Rate limits: HTTP 429 responses add the wait plus a penalty, weighted by how often a provider rate limits.
- Reliability: provider errors carry a heavy penalty. If a provider's error rate climbs far above others for the same model, it receives no requests until it recovers.
- Quality: currently mainly whether tool calls come back well formed. A provider flagged for broken tool calls, or for benchmark accuracy well below what the model should score, gets only a small amount of test traffic until the flag clears.
Failed tries are included in the cost. If 1 in 5 tries fails, the router counts 1.25 tries per good answer.
How traffic is shared
Requests are not all sent to the single best provider. Providers within a competitive range of the best Routing Cost share most of the traffic, and share drops off quickly beyond it. Three guardrails apply: every provider keeps a small floor of traffic so improvements can be noticed, a provider that starts failing or rate limiting gets less until it recovers, and a cap based on the load a provider can take before returning 429s sends the excess elsewhere. New endpoints start from the model's typical numbers. Routing remains cache-aware, so a session stays with its provider while the cache is warm.
Results and price-only routing
OpenRouter reports improvements in time to first token (16.9%), throughput (16.3%) and capacity errors (a 67% reduction), with a 6.0% increase in cost. It says this was measured before providers had time to optimize for the new algorithm. Price-only routing is still available through pinning and sorting providers.
