Live Web Search Benchmarks: Pick the Right Engine, Depth, and Model for Your Agent
By Steven Van ·
We published live leaderboards grading web search configurations across four task suites.
OpenRouter has published live leaderboards that grade web search configurations across four task suites: BrowseComp, DeepSearchQA, WideSearch, and HLE. The benchmarks compare search engines, search depth, and models by quality, cost, and speed, and update as new runs and models are added.
A few findings from the initial data:
- Search budget affects quality more than any other single factor. On BrowseComp, raising the turn budget from 1 to 25 roughly doubled scores across providers, for 2.5 to 7 times the cost per question.
- A high failure rate drives worst-case costs: models tend to keep searching up to their budget limit even when they end up wrong, so unlimited depth is costliest on hard workloads.
- Model choice matters more than engine choice. Holding the model constant, switching engine changed BrowseComp scores by about 10 points on average; switching between frontier and budget models changed scores by about 15 points.
The configurations are available today as request parameters on OpenRouter: the web plugin runs a single search before the model answers, while the server tool gives the model a search tool it can call repeatedly, up to a cap set by max_tool_calls. Engine can be set to exa, parallel, perplexity, native, or auto.
OpenRouter
One API for 500+ AI models across 80+ providers — pay with credits that work anywhere, with automatic fallback when a provider goes down.
View OpenRouter →