Web Agent Benchmarks: WebArena, WebVoyager, Mind2Web and More (2026)
By Steven Van ·
Browser Use compares sandbox, live-web and offline agent benchmarks by tasks and grading, and explains why scores from different papers rarely match.
Browser Use has published a comparison of the main benchmarks for AI agents that work on websites, sorted into three kinds. Sandbox benchmarks (WebArena, VisualWebArena, WorkArena, REAL) run on sites the benchmark controls and are mostly checked by code, so scores reproduce. Live-web benchmarks (Online-Mind2Web, WebVoyager, WebBench, AssistantBench, BrowseComp) use real sites and are graded by humans, an LLM judge or an answer key. Offline datasets such as Mind2Web score each predicted step against recorded human actions. Details on the other benchmarks come from their own papers, repos and sites, checked on October 25, 2026.
What the comparison covers
The table lists each benchmark's task type, task count, environment and grading method. Examples: WebArena has 812 tasks on self-hosted copies of a store, a forum, GitLab and a content management system. Online-Mind2Web has 300 tasks on 136 live sites. BrowseComp has 1,266 tasks that need persistent search for hard-to-find facts. Browser Use also lists its own BU Bench V2.1, marked as theirs: 200 live-web tasks, scored 0 to 100 by an LLM judge against a weighted rubric, with model cost and time reported per task. BrowserGym is included as a framework that runs several of these benchmarks in one harness, and OSWorld as a benchmark that covers the full computer rather than only the web.
Choosing a benchmark
- Reproducible research or training: WebArena, or WebArena-Verified for deterministic scoring. VisualWebArena suits agents that work from screenshots, and REAL covers replicas of consumer apps.
- Live-web reliability: Online-Mind2Web, which has a human-evaluated leaderboard, or WebBench for write tasks and logins.
- Deep research and browsing: BrowseComp, Mind2Web 2 or AssistantBench.
- Enterprise workflows: WorkArena and WorkArena++ on ServiceNow, or OSWorld if the work leaves the browser.
The post adds that none of these is the reader's own site, and that a few dozen real tasks run more than once say more than any public score.
Why scores don't compare across papers
The post gives several reasons. Harness, model and judge differ between papers: when Browser Use swapped only the judge model on the same 104 finished runs, scores ranged from 62.5% to 83.7%. Metrics differ too, since Mind2Web's paper reports macro-average accuracy. Live sites drift: Online-Mind2Web replaced 36 tasks in November 2025, and Halluminate cut WebBench to 2,454 tasks. Many results are self-reported. Browser Use notes that its own figures are, including 89.1% on WebVoyager (December 2024, 586 tasks) and 97% on Online-Mind2Web (March 2026, graded by its own judge).
The full article is on the Browser Use site.