Why latency, not intelligence, decides an AI voice call

The thing that makes an AI phone call feel wrong is almost never the answer. It is the pause before it.

Human conversational turn-taking is fast — the gaps between speakers in ordinary conversation are on the order of a couple of hundred milliseconds, and cross-linguistic research has found that range to be remarkably consistent. Anything much longer than that registers as hesitation, disagreement, or a bad line.

Where the time actually goes

A single conversational turn passes through five stages, and each one costs:

1. Endpointing — deciding the caller has finished. This is the largest and most controllable chunk. Too eager and the agent interrupts mid-sentence; too patient and every turn carries an avoidable pause.

2. Speech-to-text — transcribing what was said. Streaming transcription overlaps this with the caller still speaking; batch transcription does not start until they stop.

3. The language model — generating the reply. Time-to-first-token matters far more than total generation time, because streaming means speech can begin before generation finishes.

4. Text-to-speech — producing the audio. Again, streaming synthesis starts speaking on the first phrase rather than waiting for the whole sentence.

5. Network — every hop between the caller, the telephony provider and each service above.

What is worth optimising, in order

Endpointing configuration. Almost always the biggest available win and the one most often left at defaults. Tune it against real recordings of your actual callers, because the right setting depends on how people in your market speak, not on a benchmark.

Streaming everything. If any stage waits for the previous one to complete, the delays add up instead of overlapping. This is a plumbing decision made once and paid for on every turn.

Prompt size. A very large context on every turn costs time-to-first-token directly. Retrieving only what the current turn needs beats sending everything on every turn.

Tool calls mid-conversation. Checking a live calendar or CRM adds that system’s response time to the turn. Fetch predictably-needed data up front, and cover unavoidable lookups with a natural holding phrase rather than silence.

Model choice. A faster model that is slightly less capable usually produces a better call than a slower, smarter one — because the caller experiences the delay on every turn and the extra capability on almost none.

The trade nobody escapes

Reducing endpointing delay increases interruptions. Increasing it makes the agent feel slow. There is no setting that eliminates both, only a setting that is right for your callers.

Callers giving an address pause mid-utterance. Callers answering “yes” do not. An agent handling both well needs its endpointing tuned per expected response type — which is configuration work, not a model upgrade.

What to measure

Not average latency. Measure the distribution, and pay attention to the worst turns.

An agent averaging a comfortable delay with occasional multi-second stalls feels worse than one that is consistently a little slower, because the stalls are what callers remember and talk over.

Then measure the two things that actually matter commercially: how often callers interrupt the agent, and how far into the call they hang up. Both move with latency long before anyone complains about it, and both are recorded on every call you already have.