Agents are users, so your software needs evals
Vercel walks through how to build an end-to-end eval for software that agents use, such as an API, CLI, MCP server, or SDK. It focuses on the last step of how agents adopt a product, which is whether the agent finishes the user's task correctly. Request logs can look fine when an agent orders a hot latte for someone who asked for iced, and one good manual test can't show whether a fix worked, because LLM output varies between runs.
The guide sorts tests into three tiers by how many LLM calls they need. Deterministic tests use none, one-turn evals check a single decision such as which command the agent runs first, and E2E evals let the agent work through a whole task before checking the result. Vercel says the cheap, fast tests should still make up most of your suite.
The worked example is a made-up coffee-ordering CLI, sfbeans. The request is a large iced latte with whole milk, ready by 9:00 and under $8 including tip. Of the four cafes in the scenario, only one meets the temperature, time, and budget limits. The eval is built with Harbor, an open-source framework from the Terminal-Bench creators that runs agents in containers, checks results, and records what they did. The steps covered are:
- Pick one narrow goal, drawn from observability data, customer tickets, or your roadmap.
- Choose metrics: success rate, pass@k (can it do this at all), and pass^k (can you rely on it every time). Steps, time, and cost are measured on successful trials only.
- Build the task and run each agent setup several times against it.
The example's trial table, with Claude Code and Opus 5.5 passing three of five runs, is an illustration of what results could look like, not a measured benchmark.