# Odysseys2: a long-horizon web agent benchmark By Steven Van · 2026-10-27 Carnegie Mellon and Browser Use tested 15 agents on 100 multi-step live-web tasks, and no agent other than GPT-6 Astra fully completed more than 18. Carnegie Mellon University and [Browser Use]() have released [Odysseys2](), a benchmark of 100 long tasks on the live web. Each task is graded against pass/fail checks, 982 in total. The best agent, GPT-6 Astra, fully completes 47 of the 100. It builds on Odysseys, Carnegie Mellon's original long-horizon benchmark. ## Results A task counts as fully completed only when every one of its 8 to 12 checks is met. GPT-6 Astra (default effort) has a mean rubric score of 84.6, yet falls short on the other 53 tasks. Opus 5.5 at high effort is second with 18 tasks fully completed and a mean of 71.2. No other agent reaches 20, and 5 of the 15 agents fully complete none. - GPT-5.6 Sol: 15 tasks, mean 62.0 - Grok 4.7 (high): 13 tasks, mean 68.2 - GPT-5.6 Luna (high): 8 tasks, mean 59.7 - Fable 5, Opus 5 and Grok 4.5: 4 tasks each - GPT-5.6 Terra and Sonnet 5: 2 tasks each - GPT-5.4 mini, Gemini 3.1 Pro preview, Gemini 3.8 Flash, Gemini 3.7 Flash and Gemini Lite: none The mean score is the share of checks met, from 0 to 100. It is 0 if the attempt is flagged for suspected reward hacking. The release also reports 95% confidence intervals and median minutes per attempt. Astra's interval is 80.2 to 88.6 and its median attempt took 28.3 minutes. ## What the tasks look like The tasks are long, multi-part jobs rather than quick lookups. The median instruction is 271 words, and most ask for several deliverables. Every task starts from a Google search page, and each attempt gets 60 minutes. In the example task, the agent compares every laptop on two Amazon results pages. It is a 280-word instruction graded on 11 checks, and it asks for a workbook, a log of access failures and a written comparison. It also sets limits, such as not signing in. ## How it was run The team ran 15 agents with one attempt per task. All used BrowserCode, Browser Use's harness that drives the browser by writing code against the Chrome DevTools Protocol. Agents ran in Browser Use's stealth browsers with CAPTCHA solving. Astra ran at default effort. Opus 5.5, Grok 4.7, GPT-5.6 Luna and Gemini 3.8 Flash ran at high effort. Content filters blocked 2 of Fable 5's tasks, so its mean uses 98 tasks. Tasks were built from the real user request distribution provided by Browser Use. Source requests were de-identified and none was copied one-to-one. Anything that changes a site's state was dropped so tasks can be rerun. Four people reviewed every task and rubric, each through five rounds of model-assisted alignment and feasibility checks. ## Caveats Scores come from an LLM judge that has not been validated against a human reference. The reward-hacking flag is a model judgment, not independent proof of fabrication or intent. [![]()]() ![Browser Use]() Browser Use Open-source browser automation platform for AI agents — let LLMs drive a real browser to research, fill forms, scrape, and complete multi-step tasks. [View Browser Use →]() Sources [Original source]() [Read Odysseys2: a long-horizon web agent benchmark on Creators Toolbox]() --- Canonical source: https://creatorstoolbox.com/blog/browser-use-odysseys2-a-long-horizon-web-agent-benchmark