Voice AI Agent Development: A Complete Guide for Businesses
How to build voice AI agents: speech recognition, models, speech synthesis, turn-taking, latency, telephony, testing, compliance and Australian call rules.
Quick answer
A voice AI agent listens, decides and speaks in real time. The classic architecture is a cascaded pipeline: streaming speech-to-text, a language model with tools and business rules, and streaming text-to-speech, wrapped in turn detection and interruption handling. Speech-to-speech models collapse those steps for lower latency and more natural speech. Either way, production voice agents need a strict latency budget, telephony integration, narrow tools, clear AI disclosure, escalation to people, call recording consent where applicable and testing with realistic audio.
Where This Fits
This is the technical hub for ZSpace Labs' voice AI guides. Use cases are covered in AI voice agents for customer service, AI receptionists and AI call automation. Voice shopping is covered in ecommerce voice commerce, and general agent design in AI agent development.
Two Architectures
Cascaded (STT → LLM → TTS): separate models for recognition, reasoning and synthesis, each swappable. Text exists at every step, which helps logging, compliance checks and debugging. Latency is the sum of steps, so everything must stream.
Speech-to-speech (realtime) models: one model takes audio in and produces audio out, handling prosody and turn-taking more naturally. OpenAI's Realtime API, generally available since August 2025, supports WebRTC, WebSocket and SIP connections and tool calls. Visibility into intermediate text and voice choice can be more limited, depending on the provider.
Core Components
| Component | Job | Key choices |
|---|---|---|
| Telephony or WebRTC | Carry audio between caller and agent | SIP trunk, media streams, in-app audio |
| Voice activity and turn detection | Know when the caller has finished | Endpointing sensitivity, backchannel handling |
| Speech-to-text | Transcribe streaming audio | Accuracy by accent and domain, latency, languages |
| Language model + tools | Decide and act | Model speed, tool latency, structured outputs |
| Text-to-speech | Speak the response | Voice quality, latency, pronunciation control |
| Orchestration | Coordinate the loop, state and hand-offs | Platform or custom |
| Escalation | Transfer to people with context | Warm transfer, summary, callback |
Latency Budget
Conversations feel broken when replies lag. Break the response time into parts (network, end-of-turn detection, transcription, model time to first token, tool calls, synthesis time to first audio) and set a budget for each. Stream everything, start speaking as soon as the first sentence is ready, keep tool calls fast (or say a short holding phrase while they run), and host components close to each other. Measure latency at the 95th percentile, not just the average.
Turn-Taking and Interruptions
Callers interrupt, pause mid-sentence and say 'mm-hm'. Good voice agents stop talking when interrupted (barge-in), wait appropriately before responding, and ignore short backchannel sounds. Platforms expose settings for this; Twilio's ConversationRelay, for example, offers an interruptible setting and backchannel filtering. Tune endpointing on real calls: too eager and the agent cuts people off, too slow and it feels sluggish.
Planning a voice AI agent for your phone lines or app?
ZSpace Labs builds voice agents with streaming pipelines, fast tools, telephony integration and escalation designed in from the start.
Telephony Integration
Phone calls reach your agent through a telephony provider. Options include SIP trunks pointed at a voice platform or directly at a realtime model provider that accepts SIP, and media-streaming APIs that send call audio over WebSockets to your application. Plan for call transfer to human agents, DTMF keypad input, call recording (with consent), caller ID and number provisioning in each country you serve.
Tools, Knowledge and Guardrails
Voice agents use the same building blocks as text agents: narrow tools (look up a booking, check availability, create a ticket), retrieval for policies and FAQs, and policy checks enforced in code. Voice raises the stakes on confirmation: read back critical details (dates, amounts, addresses) before acting, and avoid long lists that are hard to follow by ear. See guardrails.
Disclosure, Consent and Compliance
- Tell callers they are speaking with an AI agent; the EU AI Act's Article 50 transparency duties apply from 2 August 2026
- Disclose call recording and obtain consent where required (rules differ by jurisdiction)
- For outbound calls in the US, the FCC has confirmed AI-generated voices count as artificial voices under the TCPA, so prior express consent rules apply
- Protect personal and payment data; avoid taking card numbers by voice unless your payment setup is designed for it
- Keep transcripts and recordings under retention rules
- Confirm obligations for your sector and markets with legal advisers
Testing Voice Agents
Test with real audio, not just text. Cover accents, background noise, poor connections, interruptions, silence, people asking for a human, wrong numbers, ambiguous dates and attempts to manipulate the agent. Measure task success, transcription accuracy on key entities (names, numbers), latency and escalation behaviour. Pilot with a small share of calls and review recordings with consent.
Advantages and Limitations
| Advantages | Limitations |
|---|---|
| Answer every call, any time, without queues | Latency and turn-taking are hard to get right |
| Handle routine requests end to end | Recognition errors on names, numbers and accents |
| Consistent information and data capture | Regulatory requirements for disclosure and consent |
| Scale for peaks | Some callers prefer people; escalation must be easy |
How to Build a Voice Agent Step by Step
- 1. Pick one call type with clear outcomes, such as booking changes
- 2. Analyse recordings or transcripts of real calls
- 3. Choose architecture: cascaded or speech-to-speech
- 4. Build fast, narrow tools and the knowledge the agent needs
- 5. Design the conversation: greeting with disclosure, confirmations, escalation
- 6. Integrate telephony with transfer and recording controls
- 7. Test with realistic audio and measure latency and success
- 8. Pilot on a share of calls, review and expand
Choosing Voice AI Components
| Component | What to evaluate |
|---|---|
| Speech-to-text | Accuracy on your callers' accents and vocabulary, streaming latency, entity accuracy for names and numbers, languages |
| Language model | Time to first token, tool-calling reliability, cost per minute of conversation |
| Text-to-speech | Naturalness, latency to first audio, pronunciation controls, voice licensing |
| Speech-to-speech model | Latency, tool support, voice options, transcript availability |
| Telephony and platform | SIP support, transfers, recording controls, regions, reliability |
Monitoring Voice Agents in Production
Track per-call metrics: end-to-end response latency per turn (p50 and p95), interruptions and talk-over events, transcription confidence on key entities, task completion, transfers and their reasons, silent periods, hang-ups mid-conversation and cost per call. Review a sample of calls each week with consent, and alert on latency spikes or rising transfer rates, which often signal a failing tool or provider issue. General agent monitoring practices are in agent observability.
Voice Agent Use Cases
| Use case | Typical tasks | Guide |
|---|---|---|
| Customer service lines | Status, simple changes, routing, summaries | AI voice agents for customer service |
| Front desk and reception | Bookings, FAQs, messages, transfers | AI receptionist |
| Outbound reminders | Appointments, deliveries, renewals with consent | AI call automation |
| Internal help lines | IT and HR questions, password resets with verification | AI customer support automation |
| In-app voice | Hands-free assistance in mobile or field apps | AI API integration |
Data Protection for Voice
Voice recordings and transcripts can contain personal, payment and sometimes health information, and a voice itself can be personal data. Decide what to record and for how long, restrict access, redact sensitive values from transcripts, avoid collecting card numbers by voice unless your payment flow is designed for it, and check where speech and model providers process and retain audio. Document these decisions in your privacy records and in what you tell callers.
Worked Example
An illustrative scenario, not a client case: a clinic group's phone lines overflow on Monday mornings. A voice agent handles appointment confirmations, cancellations and rescheduling through the booking system's API, reads back dates before changing anything, and transfers anything clinical or urgent to staff with a summary. Latency testing leads the team to cache the clinic's availability for a few seconds so the agent can answer without pauses.
Common Mistakes
- Ignoring latency until the end
- No barge-in, so the agent talks over callers
- Long, list-heavy responses
- Acting on misheard numbers without read-back
- No disclosure or recording consent
- No easy route to a human
Ready to build a voice agent callers do not hang up on?
Talk to ZSpace Labs about voice AI agent development and in-app voice experiences.
Voice Agents in Australia
Voice agents in Australia face a few specific rules and expectations. For the customer service context, see AI customer service in Australia.
- Outbound calls: telemarketing calls must follow the Do Not Call Register rules and the industry standard's calling hours (for example 9am to 8pm on weekdays, no calls on Sundays or public holidays), based on the recipient's local time.
- Call recording: recording and listening laws differ between states and territories; tell callers if calls are recorded and get legal advice on consent requirements.
- Disclosure: tell callers they are speaking with an automated agent, and offer a person.
- SMS follow-ups: marketing messages need consent under the Spam Act; factual service messages are treated differently.
- Accents and languages: test speech recognition with Australian accents and the languages your callers use; see multilingual AI.
- Privacy: recordings and transcripts are personal information; set retention and access rules.
Conclusion
Voice AI agents combine real-time audio engineering with agent design. Budget latency, handle turn-taking well, keep tools fast and narrow, disclose AI, respect consent and make escalation easy. Related: customer service voice agents, AI receptionist and AI call automation.
Common questions.
A system that holds spoken conversations: it converts speech to text or processes audio directly, uses a language model with tools to decide what to say and do, and speaks back with synthesized speech, often over phone lines or in apps.
Speech-to-text (speech recognition) converts spoken audio into text. Text-to-speech (speech synthesis) converts text into spoken audio. Voice agents use both, or a speech-to-speech model that handles audio in and out.
A cascaded pipeline (STT, LLM, TTS) gives control and choice of components and visible text at each step. Speech-to-speech models can feel more natural with lower latency but offer less visibility and voice choice. Both need tools, turn detection and safeguards.
In conversation, people notice pauses quickly. Each step (recognition, model, tools, synthesis, network) adds delay, so voice agents need a latency budget, streaming at each step and fast tools.
With voice activity detection and barge-in: when the caller speaks while the agent is talking, playback stops, and the agent processes the new input. Platforms provide settings to ignore short backchannel sounds.
Through telephony providers using SIP trunks or media-streaming APIs. Some model providers accept SIP connections directly; others connect through platforms such as Twilio.
In many places, yes or advisably. The EU AI Act requires disclosure for AI systems interacting with people from 2 August 2026 unless it is obvious, and US rules treat AI-generated voices as artificial voices for outbound calling consent. Get legal advice for your markets.
With scripted and simulated calls covering accents, noise, interruptions, silence, wrong numbers, edge cases and escalation, measuring task success, latency and transcription accuracy, plus reviewed live pilot calls.
Per-minute telephony, speech recognition and synthesis or realtime model costs, plus language model tokens and tool calls. Measure cost per handled call in a pilot.
No. Route urgent, sensitive or complex calls to people, and always provide a clear way to reach a human.
Yes. An automated agent making telemarketing calls is subject to the same Do Not Call Register rules and calling hours as a person, and call recording laws differ between states and territories. Tell callers they are speaking with an automated system, offer a human, and get legal advice on consent and recording.