When a website AI Agent disappoints, diagnose which part of the setup is failing before rebuilding it. Test the scope, source material, workflow, integrations, human follow-up, and saved configuration separately. For each issue, write down the symptom, run a repeatable test, make a targeted fix, and retest the same scenario. The worksheet at the end helps you track what you checked without treating a few tests as proof of overall performance.
1. The Agent is expected to handle too much
Symptom: It attempts to answer questions outside its intended job, or visitors are left without a useful next step when their request goes beyond that job.
Test: Write down five questions visitors might ask. Mark which ones your approved information and workflow can support. For each question outside that scope, decide what a suitable boundary response should do: ask for clarification, explain the limit, or direct the visitor to a person.
Fix: Narrow the initial job to a task you can check. For example, a home repair business might begin with questions about its published appointment policy, rather than promising to handle every possible booking change. State what is outside scope and what the visitor should do next. Retest both in-scope and out-of-scope questions. Keep additional tasks off the checklist until you can describe how each should be answered or routed.
2. A source is attached but not ready
Symptom: The Agent cannot answer a question whose answer appears in a file you supplied, or leaves out a relevant detail.
Test: Choose a specific fact in the source and ask a direct question about it. Confirm that the source has finished processing, then compare the answer with the original material. If you cannot confirm processing is complete, do not treat the missing answer as evidence that the content was indexed.
Fix: Wait until processing finishes and test again. If the answer is still missing, confirm that you supplied the intended file and that it contains the information you expect. Keep a few source-specific questions so you can repeat the check after correcting the material. Attaching a file does not prove its contents have been indexed. A source must finish processing before its content can be used in answers. For another way to test what happens when information is missing, see what your Agent should say when its knowledge has no answer. You can also review the knowledge base as you check your sources.
3. Sources conflict or describe an old policy
Symptom: The Agent gives an answer based on a previous policy, or different phrasings of a question produce answers that do not agree.
Test: Pick a question with one correct current answer. Search the material you supplied for every version of that answer. Note differences in dates, conditions, terminology, or figures. Ask the Agent the question in two ways and compare both responses with the current approved source.
Fix: Decide which material is authoritative for that subject. Correct or remove superseded information where appropriate, and make the current version clear. Retest the same question and its alternate wording. If the answer still conflicts, record the specific discrepancy before adding more material; adding another version without resolving the conflict can make the source set harder to maintain.
4. The workflow takes the wrong route
Symptom: The Agent asks an irrelevant question, skips a needed step, or starts the conversation with an opening that does not fit the visitor's request.
Test: Start a fresh preview for one common visitor request. Write down the steps you expect and note the first point where the conversation differs. Repeat with another request that should take a different route. This helps you tell a step-wording issue from a routing issue.
Fix: Correct the smallest part responsible for the wrong turn, then retest both routes. The Architect can change the text or label of one existing workflow step without rebuilding the rest of the flow. Workflow changes restart the draft preview conversation. Start a fresh preview after making a workflow change instead of continuing an earlier test. If the problem is the opening message, check where that message is supplied before editing it: The welcome text stored in appearance is a fallback. A workflow message can supply the opening message instead. See the workflow builder for workflow guidance.
5. A business integration is assumed, not connected
Symptom: The Agent can explain a general process but cannot confirm an order, update a record, or complete another request that depends on a business system.
Test: Break the visitor's request into three parts: what information must be read, what action must happen, and what result should be confirmed. Name the system needed for each part. Then check whether that information or action is actually available to the Agent. Do not treat a chat installation as proof that a separate system is connected.
Fix: Separate questions that can be answered from reference material from tasks requiring a live system connection or action. Confirm the required integration and permissions before including the task in a workflow. An embed does not itself connect order data, a CRM, or other business systems. If the connection is not in place, revise the response so it does not suggest an action has already happened; give the visitor an honest next step instead.
6. No human is available to take over
Symptom: A visitor needs help beyond the Agent's scope, but no one is clearly responsible for the handover or the visitor has no useful expectation about what happens next.
Test: Run a scenario that should go to a person. Write down who is expected to see it, what they need to do next, and what the visitor should be told while waiting. Check the path both when the responsible person is available and when they are not.
Fix: Assign responsibility for monitoring handovers. Decide what the visitor should be told when the person is unavailable, and review the process as your team's availability changes. Authorized Workspace members can take over a conversation and reply as a human. Waiting or human-takeover conversations do not receive an AI response from the widget. Human handover is a route for help, not a response-time guarantee. Set expectations your team can meet rather than implying that a handover guarantees an immediate reply. See human handover for more information.
7. A change is assumed to be a draft when it is live
Symptom: You expect to review a change later, but the field you changed affects the live Agent immediately.
Test: Before editing, write down the exact field you intend to change. After saving, check that field again and record whether it is draft or live. Avoid using “tone” as the field name: identify whether you edited wording in draft instructions or a model-context personality trait, response style, or language.
Fix: Use the saved field, not a general description of the requested change, to determine when it takes effect. Knowledge sources, the Agent name, booking links, notification recipients and model-context personality traits, response style or language are live settings and take effect immediately. Tone or personality wording inside the Agent's draft instructions waits for Publish, but changing model-context traits, response style or language does not. Check the saved field before assuming customers cannot see a change. Keep a brief change note with the field name, intended result, and next test.
Use a worksheet to find the first failing step
Run tests that matter to your business and record each result. For example, a home repair business could test whether a question about its published appointment policy receives the approved answer, then test whether a request to change a particular booking is handled within the Agent's defined scope. This is an illustrative test plan, not a report of measured results.
| Failure mode | Test question or scenario | Expected result | What happened | Next fix | Retest result |
|---|---|---|---|---|---|
| Scope | Is this request within the Agent's stated job? | Answer or clear boundary | |||
| Source readiness | Can it answer a fact in this source? | Matches the current source | |||
| Conflicting knowledge | Which policy is current? | Uses the approved version | |||
| Workflow | Does this request follow the intended steps? | Takes the expected route | |||
| Integration | Can it complete this system-dependent task? | Confirms only an available result | |||
| Human follow-up | Who receives the handover? | Clear owner and visitor expectation | |||
| Change state | Which exact field changed? | Draft or live status identified |
Count the checks you ran, the number that matched your expected result, and the failure modes that need another fix. These counts describe your checklist, not how other implementations perform. After a change, repeat the same test and expected result. If it still fails, write down what happened and choose the next specific part to investigate. This keeps the troubleshooting focused on evidence from your own setup rather than a broad verdict that the whole Agent is either working or failing.
A concrete before-and-after diagnostic
Suppose a test Agent still says “Hello” after you change the appearance welcome text to “How can our team help?” Inspect the workflow's first message. If that node still contains “Hello,” changing the fallback did not change the source of the displayed greeting.
The actual Studio screenshot uses illustrative data and shows where to inspect the selected message. Change the specific node, review its saved text, test the draft, then publish and repeat on the website. Record your observed result instead of assuming the save fixed it. The Studio review guide covers this path; knowledge readiness addresses a different cause of a wrong answer.