WTF are evals? How to test agents and the software they use

Vercel's walkthrough on evals shows how to test an agent, or the API, CLI, MCP server, or SDK it uses, by repeating a task many times and grading the outcomes. It's a guide, not a product release.

It sorts agent tests into three tiers by how many LLM calls they need. Deterministic tests use none. One-turn evals check a single decision, such as which command the agent runs first. End-to-end evals let the agent work through a whole task and check the final result. The guide says the cheap tiers should make up most of your suite. It also says "did the agent succeed?" needs the top tier, because each step can look reasonable and the order can still be wrong. Its example is a "successful" order for a hot latte when the user asked for an iced one, which looks fine in your logs.

The worked example is a made-up coffee-ordering CLI called sfbeans. The task is "a large iced latte with whole milk, ready by 9:00, under $8 including tip." The guide builds the end-to-end eval with Harbor, an open-source framework from the Terminal-Bench creators. Harbor runs agents in containers, checks results, and records what they did. Along the way it covers:

• Picking one narrow goal from observability data, support tickets, or your roadmap.
• Defining tasks, variants, trials, runs, and traces.
• Measuring success rate, pass@k (can the agent do this at all?), and pass^k (can you rely on it every time?). It also tracks steps, time, and cost on the successful trials.

The five-trial table for Claude Code with Opus 5.5 is an illustration of what results could look like. The excerpt I could see cuts off before the rest of the steps, so I can't say what the full guide does after metrics. If you're shipping something agents use, such as a CLI or MCP server, this gives you a concrete template for testing it. It's also in the Vercel listing.