# How to Test an AI Support Agent Before Launch: Test Set, Rubric, Scores By Steven Van · 2026-10-17 Quickchat's guide covers building a 30 to 50 message test set and a scoring rubric, then running both in batch on its Simulation Testing page. Quickchat has published a guide to testing an AI support agent before launch, built around a fixed test set, a written rubric and a batch run that can be repeated after every change. The [guide]() contrasts this with the common approach of asking the preview a dozen questions and launching when the answers look fine. It includes a copyable test set and rubric. ## Why spot-checking falls short The guide names three problems with testing by hand: - No memory: today's answers cannot be compared with last week's, so a knowledge base edit that fixes one question and breaks another goes unnoticed. - Selection bias: people test the questions they already thought of, while real visitors write with worse spelling and put two questions in one message. - Variation: a language model does not return the same reply twice, so one good answer proves little. ## Step 1: the test set The guide recommends 30 to 50 visitor messages across eight categories: answerable from your content, not in your content, missing information, policy boundary, pressure and injection, messy input, handoff trigger, and other language. Each category has an example message and a description of what a pass looks like. For instance, "Do you ship to Antarctica?" should get an admission that the agent lacks the information and an offer of a person, and "My order never arrived" should prompt a request for the order number. It suggests taking 20 to 30 real messages from the last month of the inbox, with names, emails and order numbers removed. It also advises putting roughly half the set into the second to fifth categories, where agents tend to break. A 16-message starter set is included in the format the Testing page accepts. ## Step 2: the rubric The Testing page scores each reply from 1 to 5 against an evaluation criteria prompt. The built-in default is a generic quality scale, so the guide suggests replacing it with one that states your own rules. Its example rubric separates hard rules (a reply that breaks one scores 1 or 2) from expected behaviour. The hard rules cover unpublished prices or policies, medical, legal or financial advice, following instructions to ignore rules, and promising refunds or discounts. Three points are given for making a rubric work: - Describe observable behaviour, such as "asks for the order number", rather than "is helpful". - Put the facts in the rubric. The judge sees the visitor message, earlier turns, the reply and the rubric, but not the knowledge base. - Use one rubric per concern. A dataset has one rubric, so separate standards need separate datasets. The criteria field accepts up to 10,000 characters. ## Step 3: creating the dataset in Quickchat Steps 3 to 5 use Simulation Testing in Quickchat, while steps 1, 2 and 6 apply on any platform that can replay a set of messages. The Testing page is in the dashboard sidebar under Conversations. A dataset is created with a name and an optional description of up to 500 characters, then filled by pasting messages separated by a line containing only --- or by uploading a CSV. For multi-column CSVs, the importer looks for a column named message, user\_message, prompt, input or text. Limits are 100 messages per dataset, 100 datasets per agent and 5,000 characters per message. A CSV with more rows than the limit is truncated. The dataset page also has an advanced Conversation metadata field that takes optional JSON, applied to every simulated conversation. It is useful when an AI Action's "Run only when" condition checks a metadata value. [![]()]() ![Quickchat]() Quickchat Build and customize AI assistants for customer support, sales, and internal use — no coding required. [View Quickchat →]() [Original source]() [Read How to Test an AI Support Agent Before Launch: Test Set, Rubric, Scores on Creators Toolbox]() --- Canonical source: https://creatorstoolbox.com/blog/quickchat-how-to-test-an-ai-support-agent-before-launch-test-set-rubric-scores