GitHub Copilot Evals is now in public preview
GitHub released Copilot Evals in public preview. It lets you write repeatable tests for the skills, custom agents, and MCP tools in your Copilot setup.
An eval gives an agent a realistic task, describes what a good result looks like, and checks both the outcome and how the agent got there. Because AI output varies from run to run, one good answer doesn't tell you much. You can rerun the same eval after changing a prompt, skill, agent, or MCP tool to catch regressions.
Ask Copilot in plain language or use the /eval command to create an eval for a customization in your project. You approve each run before it starts. When it finishes, you see which checks passed and failed. You can also ask Copilot why a check failed, and compare runs from before and after a change to see whether it helped. Under the hood, it uses Vally, a local eval tool, as its evaluation engine.
You can install it as an extension from the Visual Studio Marketplace for VS Code, as a plugin for Copilot CLI, or in the GitHub Copilot app by adding the github/evals marketplace under Customize. The preview is available on Copilot Individual, Business, and Enterprise plans.