The useful test is a short list pairing questions your approved material can answer with questions it does not cover. For each question, write down what a suitable response should include before you review the Agent’s answer. This gives you a practical way to spot gaps without treating a confident-sounding reply as proof that the information is correct.
Pick one topic and define the boundary
Choose a narrow subject, such as delivery terms, appointment preparation, or a service policy. Gather the material you want the Agent to rely on, then decide what is in scope. For example, your policy may explain standard delivery timing but say nothing about a particular delayed parcel. Write down that boundary in plain language.
For an overview of adding your material, see how to train an AI chatbot on your own data and the knowledge base guide.
Check that the material is ready
Before judging an answer, confirm that the source itself is usable. Attaching a file does not prove its contents have been indexed. A source must finish processing before its content can be used in answers. If processing is incomplete, hold off on deciding whether a missing or incorrect answer reveals a gap in the source or instructions.
Keep a note of which approved source should support each answerable question. This makes the review more specific: you can compare the response with the relevant material instead of relying on memory.
Write two kinds of test questions
Create a small list with both supported and unsupported questions:
- Supported: Ask for a detail that appears in the material. Note the expected answer and any condition that must not be lost.
- Not covered: Ask about a detail the material does not provide. Note the response standard you want, such as stating that the information is unavailable and pointing the customer to a suitable next step.
Use customer-like wording rather than copying a sentence from the source. You can also ask one supported question in two different ways. Keep each expected answer beside its question so you can compare them consistently.
Treat your standard for unsupported questions as a review criterion, not a guarantee about every response. If the Agent supplies a detail your material does not contain, mark it as unsupported and decide what needs attention before relying on that answer.
Review the responses consistently
Use the review route available in your setup, and record each result as supported, incomplete, or unsupported. For supported questions, check the important details against your notes. For questions outside the material, check whether the response stays within the boundary you set.
A useful record can be simple: the question, the source you expected it to use, the answer standard, what the response did, and the next action. This turns a vague impression into a specific task, such as clarifying a policy, adding missing information, or refining an instruction.
Make one change, then repeat the affected checks
Address one clear gap at a time, then revisit the questions affected by that change. You can ask the Architect to revise Agent instructions, appearance, and workflow through conversation. Those changes remain drafts until you publish them. Knowledge sources and model-context settings such as language and response style are live changes. Keep that draft-and-publish distinction in mind when deciding whether a change is ready to take effect.
For more on configuring an Agent, see the AI Agent builder guide. The practical outcome is a small, reusable checklist that helps you distinguish a missing source, an unclear answer standard, and a response that goes beyond the material.
Use a concrete missing-fact pair
For a fictional bakery source listing Tuesday-to-Saturday hours, “When do you close on Friday?” has a supported answer. “Can you reserve a cake for tomorrow?” does not establish a booking capability. A suitable reply explains how to contact the bakery team without claiming a reservation.
The downloadable review example includes manually authored supported, incomplete, unsupported, handover, and unresolved replies. Use them to practice the labels; they are not observed AI outputs. Then run your own questions and record the actual response separately. The no-code bakery tutorial provides the source and configuration request, and the conversation review method explains how to count the outcomes.