Instrument 08 / conversational latency

How long until the agent speaks?

Budget endpoint detection, transcript finalization, parallel fast and slow model branches, text-to-speech, and playout. Find the stage delaying the first audible response.

Enter one measured pipeline

Account for the silence before the model.

Use trace medians or percentiles from one production path. The planning bands help compare architectures; they are not universal performance benchmarks.

End-of-turn detectionSilence, semantic, or hybrid endpointing before the utterance is released. ms Transcript finalizationSpeech recognition work after endpointing until the final text is usable. ms Context and tool preparationRetrieval, memory, policy, and tool context needed by the slow branch. ms Fast model to first textTime for the low-latency branch to produce a safe opening response. ms Slow model to first textTime for the deeper branch after its context is ready. ms Text-to-speech startupTime from accepted response text to the first synthesized audio chunk. ms Transport and playout bufferNetwork delivery plus client buffering before the user hears audio. ms

Live architecture tape

See where the silence accumulates.

Parallel first audio: 900 ms. Sequential slow-model path: 1,490 ms. Parallel architecture saves 590 ms in this estimate.

Parallel response pathFirst useful branch wins

Endpoint280 ms Transcript220 ms First branch180 ms TTS160 ms Play60 ms

Swipe the tape to inspect the full path →

Sequential slow-model pathContext and deep model before speech

Endpoint280 ms Transcript220 ms Context120 ms Slow model650 ms TTS160 ms Play60 ms

Swipe the tape to inspect the full path →

Read the tape

Silence starts before inference.
Endpointing and transcript finalization may dominate the budget.

Parallelism buys first audio.
It does not make deep reasoning finish sooner.

Trace one percentile.
Do not mix median model time with tail-latency speech stages.

Transparent method

A planning model, not a production trace.

Parallel estimate. End-of-turn detection and transcript finalization happen first. Then a fast response branch races context preparation plus the slow model. The first useful text continues through TTS and playout.

Sequential baseline. The baseline waits for context preparation and the slow model before it starts TTS. The difference isolates the time-to-first-audio benefit of the parallel branch.

What is excluded. The estimate assumes stable stage timings. It does not model packet loss, jitter, barge-in recovery, speculative text cancellation, semantic quality, or an unsafe fast response.

How to use it. Enter values from the same trace percentile, compare architectures, then verify the winning design with end-to-end production instrumentation and real conversation tests.

Architecture guide

Build a voice AI latency budget that survives production.

Map the full path from the user finishing a turn to the first audible response, then decide whether a parallel fast and slow model architecture earns its coordination cost.

Read the voice AI latency budget guide →

How do you calculate voice AI response latency?

Add end-of-turn detection, transcript finalization, the first model branch to produce useful text, TTS startup, and audio playout. In a parallel design, the fast model races context preparation plus the slow model.

What is a good latency for a voice AI agent?

There is no universal cutoff. As a planning band, first audio below one second can feel conversational when turn detection is stable and the opening words are useful. Validate with production traces and interruption tests.

Why do voice AI systems use fast and slow models in parallel?

A fast branch can start a safe response while a slower branch gathers context and finishes deeper reasoning. The benefit is shorter time to first audio; the costs are coordination, handoff, and cancellation logic.

What usually causes voice AI latency?

End-of-turn detection, transcript finalization, retrieval, model startup and generation, TTS, transport, and client buffering all contribute. Measure each stage before blaming the model.