How Ora benchmarks every major AI agent on Vercel
By Steven Van ·
Ora on Vercel Front end, back end, and agent runtime on one platform Every major agent tested side by side on live sites Hundreds of commits a day from…
Ora tests whether AI agents can actually complete tasks on live websites, sending them to sign up, integrate and pay for products, and it built the whole system on Vercel, including the agent runtime, so the front end, back end and agents share the same deployment path, logs and authentication.
Ora runs each major agent, including Claude Code, ChatGPT, Gemini, Hermes, OpenClaw and Vercel's own eve framework, against customer sites and traces every step, since each harness needs its own runtime and exposes its steps differently. When eve launched, Ora benchmarked it under the same conditions as the rest: against Claude Code, using the same Claude Fable 5 and Haiku 4.5 models, across hundreds of journeys. eve came out with 7% fewer steps to reach the goal, twice the native success rate (finishing tasks on the customer's own site rather than falling back to web search), and 9% more valid endpoints. A prompt-caching fix that followed from the benchmark cut eve's total cost by roughly 15% in the next round.
Ora has since built its own agents on eve, citing eve's Next.js-style setup and a sandbox override that lets eve's agents run inside Ora's instrumented environment instead of eve's default sandbox, so they get traced like every other harness. Ora's 16-person engineering team, which ships hundreds of commits a day, also runs its infrastructure work through coding agents that operate directly against Vercel, saving what engineering lead Ido Finder estimates at a few hours a week. Ora is now splitting its platform into microservices, all deployed on Vercel, with its eve-based internal agents running as one more service. Read the full announcement.
