How to Test an AI Support Agent Before Launch: Test Set, Rubric, Scores

Quickchat has a guide to testing an AI support agent with a repeatable batch of messages and a written scoring rubric. It includes a copyable starter set and rubric.

Start with 30 to 50 visitor messages covering answerable questions, missing information, policy boundaries, injection attempts, messy input, handoffs and other languages. Write criteria around observable behaviour, such as asking for an order number.

The guide walks through running the set in Quickchat's Simulation Testing. Include the relevant facts in your rubric: the judge sees the conversation, reply and rubric, but not your knowledge base. Each dataset has one rubric, so separate concerns need separate datasets.