Editorial correction, October 4, 2026: An earlier version presented unsupported research figures as LoopReply findings. We have removed those claims and replaced the article with a review method. The numerical example below is illustrative, not customer research.
A conversation can end with a confident answer and an unresolved problem. Before you count it as a success, read what the person asked, check the answer against the source, and look for evidence that their request was handled.
You can start this review in a spreadsheet. Keep a reference to each conversation, record the outcome, and write down the next fix. The useful output is a short list of changes you can test.
Choose the question your review should answer
Decide what you want to learn before choosing conversations. For a newly launched Agent, that might be: does it answer our delivery questions correctly? For an established support workflow, it might be: what sends people to a teammate after the first answer?
Keep the scope specific enough that someone can act on the result. A review of delivery answers can lead to a clearer shipping policy. A broad score called “AI quality” can hide the exact policy sentence that needs attention.
Write the review dates and the Agent's configuration version, or the date you last changed it. If you change instructions or replace a source halfway through, separate the conversations from before and after that change.
Choose a sample you can explain
Review all relevant conversations when the volume is manageable. Otherwise, define a repeatable selection rule before reading them. For example, review the first eligible conversation from each hour of your chosen week, then keep that rule for the next review.
Record what the rule excludes. Test messages, spam, and empty widget opens can be counted separately. A conversation that ends after a difficult answer should stay in the review unless your exclusion rule has a defensible reason to remove it.
Make a second list of reported mistakes and requests for a person. These are useful for finding failures, but they are a deliberately selected group. Keep them separate when calculating a percentage for your routine sample.
Record the number of conversations you reviewed. A result from a handful of chats can tell you which question to test next; it should not be presented as a reliable forecast of all future conversations.
Read the request before judging the response
Write the customer's actual task in a few words. “Refund question” is often too broad. “Wants to know whether an opened item is returnable” gives a reviewer something concrete to check.
Then open the source the Agent should have used. Check the relevant version of the policy or product record. An answer may quote a document accurately while that document contains an outdated price.
For a numerical answer, reproduce the calculation outside the conversation. Confirm the selected sheet or dataset, the filter, and the rows included. An answer about the cheapest available item needs a comparison across the relevant records. Finding one inexpensive item does not establish that it is the cheapest.
If you cannot verify an answer, mark it as unverified and explain why. Do not turn uncertainty into a passing score just because the wording sounds plausible.
Use outcome labels with clear rules
Choose labels that your reviewers can apply consistently. This is one starting set:
| Outcome | Evidence to record |
|---|---|
| Answer verified | The answer matches the current source and addresses the request. |
| Human help needed | The request requires a teammate, approval, or an action the Agent cannot perform. |
| Answer incorrect | A statement conflicts with the source, uses the wrong record, or invents information. |
| Outcome unknown | The available conversation does not show whether the request was handled. |
Keep answer correctness separate from task completion. An Agent can explain a refund policy correctly while the refund itself still needs approval. Record both facts.
Silence is ambiguous. The visitor may have found the answer, become distracted, or given up. Without a stronger signal, an ended conversation belongs in an unknown-outcome category rather than an automatic “resolved” count.
Keep the scorecard small
For each conversation, record the topic, outcome, source checked, and the next action. Add a brief note if the answer was too long, the customer repeated a question, or the handover instruction was confusing.
A useful note names the defect: “The answer used the old 14-day policy; the current document says 30 days.” “Bad answer” leaves the next reviewer to repeat the investigation.
Give some conversations to a second reviewer. Compare disagreements and tighten the label definitions. If one person counts policy explanations as completed refunds and another does not, their resolution percentages describe different things.
Avoid building a complicated weighted score before you know how the individual labels will be used. A single incorrect account-specific answer can deserve immediate attention even when the average score looks good.
Calculate percentages with the denominator visible
Here is an illustrative example, using invented counts to demonstrate the arithmetic:
- You review 40 eligible conversations from one Agent.
- In 24, the answer matches the current source and addresses the question.
- In 8, human help is needed; 4 contain incorrect answers; 4 have an unknown outcome.
The verified-answer share is 24 divided by 40, or 60%. Describe it as “24 of the 40 reviewed conversations had a verified answer.” It is not a claim that the Agent resolved 60% of all customer problems.
If you exclude the 4 unknown outcomes, the denominator becomes 36 and the same 24 answers produce about 66.7%. The Agent has not improved. You have changed the calculation.
Keep the denominator and selection method alongside the percentage. Do not add a confidence claim unless an appropriate statistical analysis supports it.
Turn each failure into a testable change
Match the fix to the cause. If the source is missing an exception, update the source. If two documents contradict each other, resolve that conflict. If the Agent answers beyond its approved scope, clarify its instructions and test a request near that boundary.
In LoopReply, check that the relevant knowledge source is attached and ready before evaluating the answer again. A source shown in your library and a source available to the selected Agent are separate things to verify.
For structured data, keep a few rows whose answers you know. Ask for an exact match, a filtered count, and a comparison with a tie. If a value is missing, test that the response admits the gap instead of inventing a replacement.
Retest the original question and a variation
Save the failed question as a regression example. After making a change, ask it again in a fresh test conversation, then ask a natural variation with the same meaning.
Also try a question that previously worked. Replacing a policy document may fix one answer while removing information another answer relied on.
Record the expected result before you test. “It sounds better” is harder to assess than “It quotes the current return window, mentions the opened-item exception, and gives the approved contact route.”
Keep a short weekly review note with the sample dates, unresolved issues, and changes tested. Store conversation references carefully and limit copied personal information to what the review actually needs. Your next review can then check whether those specific failures still occur.
Reproduce a six-conversation review
Download the synthetic review dataset. It contains six fictional requests, replies, and review labels. No customer records or live AI outputs are included.
Using the supplied definitions, two records are supported answers, one is incomplete, one is unsupported, one is a handover, and one is unresolved. The supported-answer share is 2 divided by 6, or 33.3% rounded to one decimal. It is a property of this deliberately constructed sample, not a LoopReply resolution rate. Do not combine handovers with successful automated answers.
Review the records before looking at their labels, compare disagreements, and keep the reason for each label. Use missing-knowledge tests to repair unsupported answers and implementation troubleshooting to locate the underlying failure.