How to Write Prompts for AI Voice Agents That Actually Work
Most AI voice agent deployments fail not because the technology is lacking, but because the prompts are written by people who have never listened to a real customer call. A prompt that produces clean responses in ChatGPT will fall apart the moment a tradesman's phone rings at 7am with a panicked caller whose boiler has just packed in.
The gap between text-based AI interaction and live voice is wider than most people realise. Callers interrupt. They mumble. They call from noisy building sites or busy high streets. They ask three questions at once, then circle back to the first one halfway through your agent's response. If your prompt assumes a cooperative user who speaks in complete sentences and waits their turn, it will break within the first dozen calls.
Why ChatGPT Prompts Fail on Real Calls
Text chat gives users time to think, edit, and clarify. Voice does not. When someone calls your business, they are often multitasking, stressed, or unsure what information you actually need. They will not wait for your AI to finish a sentence before jumping in with clarification. They will not speak clearly. They will use regional terms, industry jargon, or vague descriptions that made perfect sense in their head.
Background noise is another factor that text-based testing never surfaces. A caller on a building site, in a car, or walking down a busy street will have audio that cuts in and out. Your voice agent needs to handle partial information, ask for repetition without sounding robotic, and keep track of context even when half a sentence is lost to a passing lorry.
Then there is the assumption that callers will follow a logical flow. In practice, they will answer a question you have not asked yet, skip over the question you did ask, or provide information in a sequence that makes sense to them but not to your workflow. A prompt that expects question A, then answer A, then question B will simply lose the thread when a caller opens with their postcode, their problem, and their availability all in one breath.
The Anatomy of a Voice Agent Prompt That Holds Up
A working voice agent prompt starts with a tightly defined system role. Not a generic description, but a specific instruction set that defines what the agent is, what it can and cannot do, and how it should behave under pressure. For a plumbing business, this might be: you are the first point of contact for emergency and routine callouts, you capture job details and availability, you do not quote prices, and you escalate genuine emergencies immediately.
Constraints matter more in voice than in text. You need explicit instructions on response length, because a voice agent that delivers three paragraphs of explanation will lose the caller before the second sentence. Aim for responses that take no more than ten seconds to speak. If the agent needs to convey multiple pieces of information, break it into chunks with natural pauses that allow the caller to interject.
Response format should be structured around the questions you need answered, not the questions you want to ask. If you need a postcode, a description of the problem, and a preferred time, your prompt should direct the agent to capture those three data points in whatever order the caller provides them. Use internal logic that tracks what has been gathered and what is still missing, rather than a rigid script that starts at question one and marches forward regardless of what the caller says.
Edge-case handling is where most prompts fall down. You need explicit instructions for common failure modes: callers who will not give a postcode, callers who want a price before booking, callers who are angry or abusive, callers who ask for someone by name, callers who have phoned before. Each of these scenarios needs a defined response that keeps the conversation moving without breaking the agent's role or losing the information you need.
Common Failures When Deploying Trade and Service Business Agents
Over-apologising is one of the most frequent issues we see in production. Agents default to apologetic language because most training data includes customer service scripts heavy on sorry and I understand your frustration. In practice, this makes the agent sound uncertain and encourages callers to escalate or demand a human. A confident, direct tone works better: I can get that booked in for you, or Let me take down the details and we will call you back within the hour.
Failure to capture key details happens when the prompt does not explicitly define what constitutes a complete record. For a gas engineer, that might be postcode, nature of the issue, whether there is a gas smell, and preferred contact method. If the prompt does not check for completeness before closing the call, you will end up with half-finished records that require a callback to gather the missing pieces.
Accents, jargon, and regional terms break agents that have been trained or tested only on standard speech. A caller in Hampshire might describe a problem differently than one in Scotland. Trade-specific language varies by region and by age of the caller. Your prompt needs to include examples of how people actually describe common issues, not how you describe them in your service menu. A boiler that is playing up, a tap that will not stop dripping, or a thermostat that has gone mad all mean something specific, and your agent needs to recognise them.
Testing is where theory meets reality. You cannot know if a prompt works until you run it against real calls. Synthetic testing with scripted scenarios will catch basic errors, but it will not surface the messy, illogical, interrupt-heavy conversations that make up most inbound call traffic. Deploy in a controlled environment first, listen to every call, and fix the breaks as they happen.
Writing Prompts for the Scenarios That Actually Matter
Out-of-hours calls are a common use case, but they require different handling than daytime calls. The agent needs to triage urgency, capture enough detail to decide whether someone needs to be woken up, and reassure the caller that their message will reach the right person. Your prompt should define what qualifies as an emergency, what information is needed to assess it, and what the caller should expect next. For a heating engineer, a no heat and no hot water call in January is an emergency. A radiator that is slightly cooler than the others is not.
Repeat callers are another edge case that most prompts ignore. If someone has called before, they may assume the agent knows who they are or what job they are referring to. Your system needs to handle this gracefully, either by recognising the caller and retrieving context, or by politely asking for the details again without making the caller feel dismissed. The prompt should include phrasing that acknowledges the previous contact: I have your details from last time, let me just confirm the address, or Can you remind me which job this is about so I can pull up the notes.
Pricing questions are inevitable, and most businesses do not want their AI quoting prices without seeing the job. The prompt needs a firm but polite deflection that keeps the caller engaged. Something like: Price depends on a few factors, but I can get you booked in for a free quote, or We will give you an exact price once we have had a look, no obligation. Avoid vague or evasive language that makes it sound like you are hiding something.
Handoff to a human should feel seamless. The agent needs to know when it is out of its depth and transfer without making the caller repeat everything. This means capturing enough context that the human picking up the call can see what has already been discussed. Your prompt should include a clear handoff protocol: I am going to connect you with someone who can help with that, or Let me get one of the team on the line for you. No apologies, no indication that the agent has failed.
Iterative Refinement Based on Real Call Transcripts
The first version of your prompt will not be the final version. Plan for iteration from the start. Record and review call transcripts, identify where the agent struggled, and adjust the prompt to handle those cases. Build a library of edge cases that you can test against with each update. This might include difficult accents, background noise, callers who interrupt constantly, callers who provide information in the wrong order, and callers who refuse to answer certain questions.
Look for patterns in failure modes. If the agent consistently fails to capture postcodes, the prompt may not be emphasising the importance of that field, or it may be asking at the wrong point in the conversation. If callers frequently ask for a human, the agent may be sounding too robotic or apologetic. If key details are missing from call records, the prompt may not be checking for completeness before ending the call.
Refinement is not just about fixing errors. It is about making the agent more robust, more natural, and more capable of handling the full range of situations it will encounter. This means adding examples, tightening constraints, clarifying edge-case logic, and testing each change against a representative sample of real calls. Over time, you build a prompt that does not just work in ideal conditions, but holds up under the chaotic, noisy, interrupt-heavy reality of actual customer conversations.
Most businesses underestimate the gap between a working demo and a production-ready voice agent. The difference is not the underlying model or the voice synthesis. It is the quality of the prompt, the depth of edge-case handling, and the willingness to iterate based on what actually happens when real customers call. If you are deploying AI voice agents and finding that they break under real-world conditions, the problem is almost certainly in the prompt, not the platform.
Book a free consultation to see how Antek builds and refines AI voice agents that handle your real customer calls, not just demos.