WhatsApp voice notes and AI: how an agent listens, what it gets right, and where it should hand over
In Southeast Asia a large share of WhatsApp messages are voice notes, sent from a car, a kitchen or a shop floor. A text-only bot ignores them, which is worse than no bot at all. An agent that listens, transcribes and answers treats those customers like everyone else. Here is how that works and where it should stop.
Quick answer
A modern WhatsApp AI agent transcribes each voice note, answers as if it had been typed, and keeps the transcript on the conversation so a person can read it later. Accuracy is high for clear speech in major languages and drops with noise, code-switching and dialect, so instruct the agent to confirm what it heard when the request involves money or a booking. ReplyKit transcribes voice notes on every plan; the transcription is not counted as a reply.
What happens to a voice note
- The agent receives the audio and transcribes it to text.
- It answers the transcribed request from your knowledge, exactly as it would a typed message.
- The transcript is saved on the conversation, so when a person opens the chat they read it instead of playing it.
- If the transcription is unclear or the request involves a booking, price or address, the agent repeats what it understood: "Just to confirm, you want a full service for a Honda City this Saturday morning?"
Accuracy, honestly
Clear speech in English, Malay, Chinese, Tamil or Indonesian transcribes well. Three things degrade it: background noise, switching languages mid-sentence, and heavy dialect or slang. Names and numbers are the most common errors, which is why the confirmation step matters for anything with a phone number, a plate, an amount or a date. A customer who hears their request repeated back correctly trusts the agent more, not less. Voice-note transcription is a separate small model call in ReplyKit and is not counted against your reply allowance.
Photos and videos
Photos get the same treatment: the agent describes what it sees (a tyre, a rash, a floor plan, a menu) and answers within the boundaries you set. For a workshop that means "I can see a warning light, let's book an inspection"; for a clinic it means "thanks for the photo, the doctor will assess it at your consultation". It should never diagnose or promise a result from an image, and you write that rule in the instructions. Videos are acknowledged and handed to a person; the agent does not watch video.
When a voice note should go to a person
- The transcription confidence is low and the customer's request is not clear after one confirmation.
- The voice note is long and emotional, which usually means a complaint.
- It contains details the agent must not act on: card numbers, medical history.
- The customer asks for a person, in any language.
The handover message is the same as for text: a person will reply here shortly. See how handover should work.
Frequently asked questions
Can an AI agent understand WhatsApp voice notes?
Yes. ReplyKit transcribes each voice note and answers it like a typed message, keeping the transcript on the conversation.
Does it work in Malay and Chinese?
Yes, for clear speech. Accuracy drops with noise, dialect and mid-sentence language switching, so the agent confirms details that matter.
Are voice notes counted as replies?
The transcription is not counted. Only the agent's reply counts against the monthly allowance.
What about photos?
The agent describes images and answers within your boundaries, for example booking an inspection rather than diagnosing.
Can it watch videos?
No. Videos are acknowledged and handed to a person.
Send the demo a voice note
Say what you want in Malay or English and watch it answer. Then set up your own. 7-day free trial.
Sources and further reading
- DataReportal, Digital 2025: Malaysia · WhatsApp is the most used messaging platform in Malaysia
- WhatsApp Business Messaging Policy · opt-in, opt-out and prohibited-use rules every business must follow
- Harvard Business Review, The Short Life of Online Sales Leads (Oldroyd, McElheran, Elkington, 2011) · firms that responded within an hour were about seven times more likely to qualify the lead
ReplyKit is an independent product and is not affiliated with or endorsed by Meta or WhatsApp. This article is general guidance for running a small business, not legal, financial or regulatory advice.