Voice AI: real-time voice agents
04
Key concepts
9 min read · Chapter 4 of 10
01
Entity and identity
What is the unit of record, and how do we know it is the same one?
Three identities meet on every call: the business the agent speaks for, the voice (synthetic, cloned or human) and the phone number. The caller's own number can be spoofed. STIR/SHAKEN attestation, when present, says how much the originating carrier vouches for it: an A means the carrier knows its customer may use the number, and says nothing about the voice or the words.
Inside the stack, one call collects IDs from the carrier, the communications platform, the voice platform, the model session and the trace. Joining them is real engineering work, and you need it before you can debug a bad call or reconcile a bill.
The law names its own entities, and one platform can be several at once:
| Legal role | Law | What it triggers |
|---|---|---|
| The "maker" or initiator of a call | TCPA | Consent, caller identification, opt-out |
| A third party listening in | California's wiretap law | $5,000 per violation if callers weren't told |
| Provider or deployer | EU AI Act | Watermarking for the provider; disclosure for both |
| Business associate | HIPAA | A BAA, and BAAs with every sub-vendor |
| A collector of voiceprints | Illinois's biometric law (BIPA) and others | Written consent and a retention policy |
A voiceprint is a biometric template that identifies a speaker. A plain transcription agent generally doesn't build one, though courts are testing that line in cases about meeting transcription tools.
02
State and lifecycle
What states exist, and what moves an entity between them?
A call carries several state machines that move at different speeds.
- Turn: listening, caller speaking, end of turn pending (when the model starts early), thinking, tool running, agent speaking, interrupted, then back to listening.
- Call: ringing, answered by a person or a machine, in progress, transferring, ended (completed, no answer, busy, failed or cancelled).
- Model session: Google's Live API adds connected, "going away" (about 60 seconds' warning) and resumed, because a connection lasts about 10 minutes and an audio session about 15.
- Consent: none, prior express, prior express written, then revoked, and a revocation has to be honoured within 10 business days.
On outbound calls, answering-machine detection takes about 4 seconds after pickup with Twilio's defaults. A model upgrade changes turn-taking behaviour as well as answers, so I'd put timing in every regression test.
03
System of record and ledger
Who owns the truth, and how do systems reconcile?
Four records describe one call, and they disagree by design.
| Fact | System of record |
|---|---|
| Minutes billed for the call | The carrier's and communications platform's call detail records |
| What the agent did, turn by turn | The voice platform's call log |
| What was said | The transcript and the recording |
| What changed in the business | The tool-call log in the business's own systems |
| Whether the caller agreed to be called | The deployer's consent records |
Two details matter. In a speech-to-speech model the transcript is a side product of a separate recognizer, so an audit or eval built on transcripts can miss what the model actually "heard". And after an interruption, what the model thinks it said and what the caller heard diverge unless the history was cut back.
Consent records are the defence in every TCPA case, and the platforms say plainly that they don't keep them for you: Retell's terms say it "does not obtain consent on behalf of Customer". Billing is its own reconciliation job, since five vendors can meter one call in carrier minutes, stream time, audio tokens, characters and voice minutes billed by the second.
04
Rules and policy
What logic decides outcomes, and who can change it?
Turn-taking settings are policy decisions: how long a silence ends a turn, how eager the model is to answer, how long a sound has to last before it counts as an interruption. A longer timeout while the caller reads out digits protects accuracy and costs speed.
Guardrails behave differently by architecture. A chained stack can check the text before it's spoken. On a speech-to-speech stream the check runs alongside the speech, and an open issue on OpenAI's agent toolkit reports the agent kept talking after a guardrail tripped. Required disclosures and consent prompts are fixed text inside a probabilistic stream, and a chained or hybrid stack makes it easier to guarantee they're said word for word.
Above the builder's settings sit laws, card network rules, carrier analytics and the platform's and model provider's usage policies. Retell's terms, for example, require outbound agents to identify themselves and ban calls to emergency lines.
05
Effective dating
Which version of the rule applied at that moment?
Dates I'd keep on a calendar as of October 2026:
| Change | Effective | Status (Oct 2026) |
|---|---|---|
| FCC: AI voices are "artificial" under the TCPA | February 8, 2024 | In force, and applied to calls made before it in pending suits |
| California: prerecorded calls must say if the voice is artificial | January 1, 2025 | In force |
| Maine: AI chatbots, spoken ones included, need a clear notice | September 2025 | In force |
| Carriers must sign caller ID with their own token | September 18, 2025 | In force |
| Carriers blocking on analytics must return a specific SIP code (603+) | March 25, 2026 | In force |
| EU AI Act: tell people they're talking to AI; mark synthetic audio | August 2, 2026 | In force |
| FCC comment deadline on letting political AI calls reach mobiles without consent | October 19, 2026 | Upcoming |
| EU: marking deadline for audio systems already on the market | December 2, 2026 | Upcoming |
| Google's Gemini Flash and Flash speech models double in price | January 1, 2027 | Upcoming |
Some dates float. Deepgram's streaming prices are "limited-time promotional rates" with no end date, so I'd budget at the regular rate. Google's Live API is a preview with no deprecation guarantees, and some OpenAI model names point at a default snapshot that can change, so I'd pin dated snapshots where offered. The FCC's 2024 proposal to require AI disclosure at the start of calls hasn't been adopted, and Canada's regulator opened its own review in June 2026.
06
Interfaces and standards
What format and protocol do counterparties speak?
The phone side is standardized: SIP for setting up and transferring calls, RTP and SRTP for the audio, G.711 at 8 kHz on the phone network, Opus in apps, E.164 numbers, STIR/SHAKEN for caller ID, and DTMF for keypad tones. The AI side has no shared standard. Every platform has its own WebSocket event protocol (Twilio's media streams, OpenAI's Realtime events, Google's Live messages, Deepgram's turn events), and nothing covers turn events, barge-in or tool calls across vendors. That's a lock-in point, and the reason open frameworks like Pipecat and LiveKit's agents exist.
WebRTC
- Typical use
- A browser or app talking to a model or platform
- Strengths
- Built for live audio: handles loss, jitter and echo; OpenAI's recommended client path
- Weaknesses
- Needs media servers and network traversal
WebSocket media stream
- Typical use
- A communications platform sending call audio to your server
- Strengths
- Simple, works everywhere
- Weaknesses
- Lost packets stall the stream; you handle buffering and barge-in
SIP trunk to a platform or model
- Typical use
- Phone numbers into LiveKit, OpenAI or a voice platform
- Strengths
- Fewer hops; native transfers; carrier-grade
- Weaknesses
- You still need a trunk provider; caller ID can change on transfer
The phone network end to end
- Typical use
- Any phone
- Strengths
- Reaches everyone
- Weaknesses
- 8 kHz audio, extra delay, spam labels and blocking
OpenAI now accepts SIP calls directly, with a separate EU endpoint, and warns builders not to retry an outbound call automatically after an ambiguous timeout, because the retry can place a second call. For provenance, Google's SynthID and Meta's AudioSeal watermark synthetic audio and C2PA labels files, but a watermark only marks the provider's own output, and phone audio degrades it.
07
Networks and counterparties
Who sits between us and the outcome, and what do they want?
A call passes through the caller's carrier, transit carriers, a communications platform or SIP trunk, the voice platform, one to three model vendors (often on different clouds) and the business's own systems. Twilio counts at least ten network traversals per turn of a chained agent, and each boundary can add another encode, decode and buffer. Moving the pieces closer together is mostly about latency: Telnyx places GPUs next to its telephony sites, and speech-to-speech models collapse three vendors into one.
| Party | What they control | What they earn |
|---|---|---|
| Communications platform or carrier | Numbers, caller-ID attestation, the audio stream | Per-minute and per-number fees, plus carrier surcharges |
| Terminating carriers and analytics engines (Hiya, TNS, First Orion) | Spam labels and blocking | Branded-calling services sold to businesses |
| Voice platform | The turn loop, logs, transfers | A per-minute platform fee |
| Model vendors | Accuracy, voices, latency, retirements | Per token, minute or character |
The analytics engines decide whether an outbound agent is heard at all. An outbound agent dials many short calls from few numbers, which is the pattern that earns a "Spam Likely" label, and a caller-ID company's 2026 survey found 86% of consumers say they won't answer an unidentified call (a vendor survey). Registration and branded calling are in Phone numbers, messaging and voice.
08
Regulatory layering
Jurisdiction × activity × entity type: is it a license or a certification?
| Call | Layers that apply |
|---|---|
| Outbound, US | TCPA and the FCC's rules, state telemarketing laws, state AI-disclosure laws, state recording laws, carrier policy, the platform's terms |
| Inbound, US | State AI-disclosure laws (Maine, Utah), recording and wiretap laws, biometric laws if voiceprints are used |
| Any call in the EU | AI Act Article 50, GDPR, national telemarketing and recording rules |
| Health or card data, anywhere | HIPAA and BAAs; PCI DSS through the card networks |
| Canada | The CRTC's rules on automated calls (which already cover synthesized voices), privacy law |
One agent usually serves callers in every state, so in practice the strictest layer wins: an all-party recording notice on every call, AI disclosure in every greeting.
09
Exceptions and reversals
What goes wrong, and how is it undone?
Most reversals on a call are small and happen every minute: a speculative reply is cancelled when the caller keeps talking, the agent's turn is cut back after an interruption, a failed transfer comes back to the agent. The dangerous ones are side effects. A tool call cancelled mid-flight because the caller changed their mind may already have done something, and I couldn't find how vendors handle that. An outbound dial retried after an ambiguous timeout can ring the same person twice, a second TCPA exposure.
Caller revokes consent
- Who starts it
- The caller, by any reasonable means
- Clock
- Honour within 10 business days
- The way back
- Suppress the number everywhere
Transfer fails
- Who starts it
- Trunk, carrier or an unanswered human
- Clock
- Seconds
- The way back
- Return to the agent; offer a callback
Model or price change
- Who starts it
- The vendor
- Clock
- Days to months; previews without notice
- The way back
- Pinned snapshots; regression tests with timing
Vendor shuts down after an acquisition
- Who starts it
- The acquirer
- Clock
- Weeks: PlayHT's API went dark in July 2025
- The way back
- Re-integrate; cloned voices may be lost
Deployment pulled back
- Who starts it
- The business
- Clock
- After the damage
- The way back
- Rehire, apologize, rethink scope
Commercial terms reverse too. Bland raised its per-minute price by 22-56% in December 2025, and ElevenLabs said in February 2025 that it was absorbing language model costs and would eventually pass them on.
10
Liability allocation
When it fails, who pays?
The deployer owns what its agent says. When Air Canada's chatbot invented a bereavement-fare rule in 2024, the tribunal rejected the airline's argument that the bot was a separate legal entity and made it pay about C$812, and the same principle applies to a voice agent that quotes a refund on a recorded call.
Platform contracts push the rest down to the customer:
| Clause | Retell (June 2026) | Vapi (September 2026) |
|---|---|---|
| Who gets consent | The customer only | The customer |
| AI disclosure | Required at the start of outbound calls | Not explicit |
| Customer indemnity | Covers regulatory fines, the TCPA included; uncapped | Covers TCPA claims |
| Vendor's cap | 12 months of fees | The greater of $100 or 12 months of fees |
| Accuracy | Not warranted; the customer assumes the risk | All warranties disclaimed |
Plaintiffs are testing whether that holds. A December 2025 suit claims OpenAI and Twilio are liable as makers of AI robocalls that another company sent through them, two California rulings in 2025 let wiretap claims proceed against AI vendors able to use call audio for their own models, and nine biometric suits filed in May 2026 accuse ElevenLabs and big tech companies of building voiceprints from public recordings to train models. An indemnity doesn't stop a plaintiff from suing the vendor directly; it only moves the bill, and only if the customer can pay.
Carriers carry their own share through attestation: Lingo Telecom paid $1 million for signing spoofed AI calls. Fraud losses depend on the payment rail and the country. In the US the victim of a scam that tricks them into sending money usually bears it. Pindrop reportedly offers a deepfake warranty of up to $1 million a claim, the only explicit transfer of this risk from a security vendor I found.