Run Terminal-Bench and other Harbor evals on Vercel Sandbox
By Steven Van ·
Harbor's Terminal-Bench, SWE-bench and other evals now run on Vercel Sandbox, with each trial in its own Firecracker microVM.
Harbor, the open-source evaluation harness built by the Terminal-Bench team, can now run its trials on Vercel Sandbox. Passing --env vercel to harbor run puts each trial in its own isolated Firecracker microVM instead of a local Docker container, so many more trials can run in parallel than on a local machine. The integration covers the whole Harbor Hub registry, more than 300 datasets including Terminal-Bench, SWE-bench, tau3-bench and OSWorld, not just Terminal-Bench itself.
- A task's network policy (public, no network, or a hostname allowlist) is enforced at the sandbox firewall, outside the VM.
- Optional credential injection attaches secrets to matching outbound requests at that firewall, so they never enter the sandbox.
- Paired with AI Gateway, a single AI_GATEWAY_API_KEY reaches hundreds of models from multiple providers, so benchmarking a different model is the same command with a different --model value.
The feature requires Harbor 0.22.0 or later. Full setup, configuration and troubleshooting steps are in the step-by-step guide.
Vercel
The platform for frontend developers — deploy, preview, and scale web apps and AI agents with zero config.
View Vercel →