Diagnosing a wrong answer from an AI agent: the six causes (outdated knowledge, conflicting lines, ambiguous wording, missing knowledge, an instruction gap, a retrieval miss), how to tell which one it was from the conversation, and the one-line fix for each
The agent told a customer the wrong price, or the wrong opening time, or that you offer something you do not. The first reaction is to distrust the agent. The useful reaction is to open the knowledge base, because that is where the cause almost always is, and the fix is a line.
Quick answer
Wrong answers come from six causes: outdated knowledge (the old price is still there), conflicting lines (two sources disagree), ambiguous wording ("open late"), missing knowledge (the agent guessed instead of saying it did not know, which should be rare), an instruction gap (nothing told it to hand that topic over), or a retrieval miss (the right line exists but did not match the question's words). Read the conversation, search the knowledge base for the topic, and the cause is usually obvious. Fix the text, retest in the sandbox, and move on. ReplyKit shows which knowledge the agent used for each reply, which shortens the diagnosis.
The six causes
| Cause | How it looks | Fix |
|---|---|---|
| Outdated knowledge | The agent quotes a price or hour you changed | Delete the old line; keep one version of every fact |
| Conflicting lines | Website says one thing, pasted FAQ another; the agent picked one | Merge; the knowledge base should have one source per fact |
| Ambiguous wording | "Open late", "from $50", "usually" | Rewrite with numbers and conditions |
| Missing knowledge | The agent answered a question nothing covered | Add the fact; if it guessed rather than saying "not sure", tighten the instruction to prefer "not sure" |
| Instruction gap | It answered something you wanted handed over (a medical question, a custom quote) | Add the hand-over rule |
| Retrieval miss | The right line exists but the customer used different words or a different language | Add the customer's words or a glossary term next to the fact |
Diagnosing in two minutes
- Read the conversation: exactly what did the customer ask, and what did the agent say?
- Search the knowledge base for the topic. Two hits that disagree: conflict or outdated. One ambiguous hit: wording. No hit: missing or retrieval.
- Check which knowledge the agent used for the reply, shown alongside it. If it used an unrelated line, the match was weak: retrieval.
- Check the instructions: should this have been a hand-over?
- Fix the text. Retest with the customer's exact words in the sandbox.
The retrieval mechanism, and why wording matters, is in how retrieval works; the reason guesses are rare is in why agents hallucinate.
Preventing the next one
- One source per fact. When a price changes, change it in one place and delete the rest; the structure is in what to put in the knowledge base.
- Numbers and conditions, never adjectives.
- A glossary of customer words for your terms, especially in other languages; see multilingual knowledge.
- Re-crawl the website after you change it, and check the crawl replaced the old version.
- A weekly ten minutes on the gap list and a monthly read of a sample of conversations; the routine is in the monthly audit.
What to do about the customer
Message them, correct it, and honour the wrong answer if it was a price you can bear; it was your text that was wrong. The service-recovery wording is in apology and service recovery templates. Then fix the line so it does not happen to the next one.
Frequently asked questions
Why did my AI agent give a wrong price?
Almost always because the old price is still in the knowledge base alongside the new one, or the wording was ambiguous. Delete the old line.
How do I find the cause?
Read the chat, search the knowledge for the topic, and check which knowledge the agent used for the reply. The cause is usually obvious.
Is the agent itself faulty?
Rarely. The fix is nearly always in the text: a line removed, merged, rewritten or added.
Why did it guess instead of saying it did not know?
The instruction to prefer "not sure" may be too weak for that topic; tighten it and add the fact.
Should I honour a wrong price?
If you can bear it, yes; it was your text. Then fix the text.
Fix the line, not the agent
See which knowledge each reply used, fix the text, retest in seconds. 7-day free trial, no card.
Sources and further reading
- Harvard Business Review, The Short Life of Online Sales Leads (Oldroyd, McElheran, Elkington, 2011) · firms that responded within an hour were about seven times more likely to qualify the lead
- WhatsApp Business Messaging Policy · opt-in, opt-out and prohibited-use rules every business must follow
ReplyKit is an independent product and is not affiliated with or endorsed by Meta or WhatsApp. This article is general guidance for running a small business, not legal, financial or regulatory advice.