How it works

How one turn of a call works, step by step

Here's one turn of that payroll call, on a chained pipeline, from a mobile phone, using Twilio's target timings. The hardest decision often comes at the very start: deciding the caller has finished speaking, without cutting them off or leaving a long silence. Then the agent answers or calls a tool, hands off to a person when it should, and stops talking when the caller interrupts.

1. Caller speaks: "My payroll didn't go out on Friday." The audio streams in across the mobile network and the carriers to the communications platform's edge, where it's buffered and decoded (about 95 ms in Twilio's diagram). Phone audio is narrowband G.711 at 8 kHz, which carries roughly 300-3,400 Hz, so a model that wants 16 kHz audio gets a resampled signal with nothing above about 3.4 kHz. What can go wrong: background noise, a TV or another person talking; one noise-cancellation vendor claims background voices push transcription errors from 5% to over 30%.
2. Turn ended: the caller pauses, and the platform has to decide whether they're done. Voice activity detection (VAD) only says whether someone is speaking; end-of-turn detection decides they've finished. A plain silence timer waits about 500 ms by default. Turn models also weigh the words and the intonation: Deepgram claims about 260 ms for its Flux model, and Pipecat's open Smart Turn model runs in about 10 ms on some CPUs. Some stacks start the language model early on a likely end of turn and cancel if the caller keeps going, which saves hundreds of milliseconds and costs 50-70% more model calls (Deepgram's estimate). What can go wrong: too eager and the agent cuts off a caller reading out an account number; too patient and the line goes quiet. On τ-Voice, realistic turn-taking cut retail task success by 7 points overall and by 11 for Google's model.
3. Understood: a streaming speech-to-text model sends partial transcripts and then a final one, with a target of 350 ms and an upper limit of 500. In a speech-to-speech model the audio goes to the model directly. What can go wrong: names, emails, addresses and strings of letters and digits fail most, and OpenAI listed "better alphanumeric recognition" as an improvement in gpt-realtime-2.1. A 2020 study of five commercial recognizers found almost twice the word error rate for Black speakers as for white speakers, and τ-Voice found accents cut task success by 10 points.
4. Answered: the language model chooses a reply or calls a tool. Time to first token is the largest and most variable slice of the budget, with a target of 375 ms and an upper limit of 750. A tool call, such as looking up the payroll run, adds the tool's round trip and another model call, and the tool definitions and the history are re-sent every turn. What can go wrong: a slow backend leaves dead air; a speech-to-speech model promises something before the tool returns; a guardrail that checks a transcript trips after the words are spoken.
5. Spoken: text to speech streams the first audio chunk before the sentence is finished. Twilio targets 100 ms; Vapi's tests in June 2026 measured 159-197 ms for the fastest voices, network included, about twice what vendors quote for the model alone. Add about 10 ms for each of roughly eight hops between services and 40-100 ms back to the caller's ear. What can go wrong: the 95th-percentile turn is much slower than the median, and a dashboard that leaves out end-of-turn time and the network (Vapi's own estimate does) understates what the caller hears.

Two exits sit off that path. Handed off is the move to a person. A cold transfer sends a SIP REFER through the trunk and the agent drops out; a warm transfer puts the caller on hold, briefs a human, connects them and leaves, and comes back to the caller if nobody answers. The payroll agent hands off whenever someone asks to change bank details. Transfers fail when the carrier rejects the REFER, when the caller ID shown to the human changes unexpectedly, or when a cold transfer loses the context and the caller starts over.

Interrupted is the caller talking over the agent, called barge-in. Stopping takes two jobs: flush the audio queued for playback (Twilio's clear message), then cut the agent's turn in the history down to what the caller actually heard (Twilio's mark messages report what played). Skip the second job and the model believes it said things the caller never heard. The opposite failure is a false stop on a cough, a "mm-hmm" or a TV. τ-Voice also measured interruption rates under realistic audio, 14% for OpenAI's model, 21% for Google's and 84% for xAI's, which shows how far apart the models still are on turn-taking.

The stages overlap, so they don't add up exactly. On a phone call, the network alone takes a real share: the international telecom standard treats one-way delay under 150 ms as unnoticeable, and mobile networks are designed to about 200 ms one way.

Chained, speech-to-speech or hybrid

A chained stack (also called cascaded, or a pipeline) runs speech to text, then a language model, then text to speech. Every hop is text you can read, log and check, and you can swap any part. OpenAI's own guidance says to choose it when you need to inspect or transform the text in the middle, for example to run a policy check before the agent speaks.

A speech-to-speech model (sold as "realtime" or "native audio") hears and speaks in one step and keeps the tone that a transcript drops. It isn't automatically faster: Artificial Analysis measured 0.44-0.82 seconds to first audio at low reasoning and 2.3-3.0 seconds at high reasoning, and τ-Voice measured 0.90-1.15 seconds under phone-like audio, the same range as Twilio's chained target. I found no independent test of the two on the same tasks and audio, which is the biggest gap in the evidence. Cost grows with call length, because OpenAI's Realtime API re-sends the whole conversation with each response. By my arithmetic from its prices, five minutes into a call an uncached session costs $0.77-1.15 a minute, against about a cent when the cache works.

A hybrid splits the fast talker from the slow thinker. OpenAI's GPT-Live-1 is full-duplex (it listens while it speaks) and hands reasoning and tool calls to a separate backend model, billed separately; OpenAI reports it cut turn-taking latency to 0.8 seconds from 1.41, a figure I saw only in press coverage. NVIDIA's open research model speaks an on-hold message while a tool runs, and LiveKit, Pipecat and Deepgram start the language model speculatively inside chained stacks.

My default: chained or hybrid for anything with a regulated script, a payment or a policy the agent must quote exactly, and speech-to-speech where tone matters more than exact words and calls are short. Whatever the architecture, I'd measure mouth to ear on real phone calls at the 95th percentile.