Hallucination, explained: why AI invents prices and policies, and the four guards that stop it in a customer service agent
A language model is built to produce plausible text, not true text. Ask it a price it does not know and it will produce a plausible one. In casual use that is a curiosity; in customer service it is a refund, a lost customer or a legal problem. The fix is not a better model. It is a design that never lets the model answer from nothing.
Quick answer
Hallucination is when an AI states something false with confidence, because it is generating plausible text rather than looking anything up. In customer service that means invented prices, made-up policies and promised appointment slots that do not exist. A well-designed agent prevents it with four guards: it answers only from your knowledge base, it says it is not sure when nothing matches, its scope is limited to your business, and it hands over to a person for anything it cannot ground. ReplyKit is built with all four.
Why it happens
A language model predicts the next words from patterns in text. It has no built-in notion of whether a sentence is true, only whether it is likely. "A consultation costs RM 120" is a perfectly likely sentence, so a model asked for a consultation price with nothing to go on will produce one. Fluency is the whole problem: the false answer sounds exactly like the true one.
What it does in customer service
- A quoted price you have to honour or argue about.
- A policy the customer relies on ("refunds within 30 days") that does not exist.
- A slot promised that is not free, and a customer who turns up.
- Medical or legal advice that no one licensed gave.
- A claim about a competitor or a product that is simply false.
Every one of these is said in your name, which our terms are explicit about.
The four guards
- Answer only from the knowledge base. Every reply is written from passages retrieved from your documents. No passage, no answer. See how retrieval works.
- Admit uncertainty. When nothing matches, the agent says it is not sure and offers a person, and logs the question. See why the agent says it is not sure.
- Limit scope. The agent is your business's assistant, not a general assistant. It refuses to be anything else, and it refuses regulated advice regardless of instructions.
- Hand over. Complaints, money, medical, legal and anything ambiguous go to a person. The AI never gets the last word on what matters.
How to check your agent
Ask it a price not on your list, a policy you do not have, and a slot on a day you are closed. A guarded agent says it is not sure or offers what actually exists. An unguarded one answers all three confidently. The twenty-message test includes these checks.
Frequently asked questions
What is AI hallucination?
When an AI states something false with confidence, because it generates plausible text rather than checking facts.
Why would a chatbot invent a price?
Because a price is a likely thing to say when asked for one, and the model has no way to know it is wrong unless it is restricted to your documents.
How does ReplyKit prevent hallucination?
It answers only from your knowledge base, admits uncertainty, limits scope to your business, and hands over to a person for anything it cannot ground.
Can hallucination be eliminated completely?
The guards make invented facts very rare and make the failure mode a handover rather than a wrong answer. Review conversations monthly to catch anything that slips.
How do I test for it?
Ask for a price, policy or slot that does not exist. A guarded agent says it is not sure.
Truth by design
Answers only from your knowledge, honest about the rest. 7-day free trial.
Sources and further reading
- ReplyKit Terms of Service, section 4 · statements by the agent are made on your behalf
- ReplyKit Acceptable Use Policy · what we refuse to automate
- Harvard Business Review, The Short Life of Online Sales Leads (Oldroyd, McElheran, Elkington, 2011) · firms that responded within an hour were about seven times more likely to qualify the lead
ReplyKit is an independent product and is not affiliated with or endorsed by Meta or WhatsApp. This article is general guidance for running a small business, not legal, financial or regulatory advice.