The 30-minute monthly conversation audit: what to read, what to count, and what to change
An AI agent is only as good as the last time someone read what it said. A monthly audit takes half an hour, catches drift before customers do, and produces exactly one change to make. Here is the routine.
Quick answer
Once a month, read twenty conversations (ten random, five handovers, five from the unanswered list), score each on five questions (accurate, on-tone, asked the right question, handed over correctly, booked when it could), count handover rate, unanswered questions and booking rate, and make one change to instructions or knowledge. Write down the numbers so next month has a baseline.
The sample
- Ten conversations at random from the month.
- Five that were handed to a person, from the inbox filter.
- Five from the unanswered-questions list.
- Any that ended in a complaint.
Five questions per conversation
- Accurate? Every fact matches the knowledge base. A wrong price is a knowledge fix.
- On tone? Sounds like the business. Too formal or too chatty is an instructions fix.
- Asked the right question? Moved the customer toward booking or a clear next step.
- Handed over correctly? Not too early (could have answered), not too late (argued or improvised).
- Booked when it could? A customer who wanted a slot got offered one.
Mark each yes or no. Twenty conversations, five questions, a hundred marks in twenty minutes.
Three numbers
From the leads page and the digest: handover rate (handovers divided by conversations), unanswered questions logged, and booking rate (booked divided by new leads). Write them in a note with last month's. A rising handover rate means missing knowledge; a rising unanswered count means a new kind of question; a falling booking rate means the offer or the slots need attention. The response-time method can be added quarterly.
One change
Pick the most common "no" from the scoring and fix it: a line in the instructions, a document in the knowledge base, a teach session on the unanswered list. One change a month is enough; several at once make it impossible to know what worked. Re-run the twenty-message test after an instructions change. The mistakes guide maps common findings to fixes.
Frequently asked questions
How often should I review my AI agent's conversations?
Monthly, twenty conversations, half an hour. Weekly for the unanswered-questions list.
What should I look for?
Accuracy, tone, the right question, correct handover, and whether it booked when it could.
Which numbers matter?
Handover rate, unanswered questions logged, and booking rate, compared with last month.
How many changes should I make?
One a month, so you know what worked.
Where do I find handovers and unanswered questions?
The inbox filter for waiting chats and the unanswered list on the Agents page; the Monday digest summarises both.
Half an hour a month keeps it honest
Inbox filters, unanswered list and a digest with the numbers. 7-day free trial.
Sources and further reading
- Harvard Business Review, The Short Life of Online Sales Leads (Oldroyd, McElheran, Elkington, 2011) · firms that responded within an hour were about seven times more likely to qualify the lead
- WhatsApp Business Messaging Policy · opt-in, opt-out and prohibited-use rules every business must follow
ReplyKit is an independent product and is not affiliated with or endorsed by Meta or WhatsApp. This article is general guidance for running a small business, not legal, financial or regulatory advice.