Semantic turn detection

Semantic turn detection is deciding when a caller has finished speaking based on whether their words sound complete, rather than only on how long the silence has lasted. It is also called semantic VAD, positioning it as the linguistic layer above acoustic voice activity detection, which can tell speech from silence but has no idea whether a sentence has ended.

The companion metric is endpointing latency: the time from the true acoustic end of the customer's utterance to the agent's decision to respond. That number is where voice agents earn or lose their credibility. Callsphere's March 2026 analysis reports that humans take turns at roughly 200 to 300 millisecond gaps, while most voice agents lag at 800 to 1500 milliseconds because they wait out a fixed silence threshold, and that semantic turn detection combining audio and text closes the gap to around 300 milliseconds without cutting users off mid-thought.

Why silence-based turn detection fails

Classical endpointing waits for a fixed duration of silence and then declares the turn over. The threshold is a single number, and every value of it is wrong for some real conversation.

Set it short and the agent is fast and rude. It fires during the pause a caller takes while reading a number off a card, then talks over them. Set it long and the agent is polite and dead. The caller finishes a short question, waits through a silence that feels like a dropped call, and often starts repeating themselves just as the agent begins to speak. Neither setting is a tuning failure. The trade-off is structural, because silence duration carries almost no information about whether a thought is finished.

Human conversation does not work this way. People project the end of a turn before it arrives, using syntax, prosody, and pragmatic completeness, which is how turn-taking happens at gaps far shorter than any usable silence threshold. An agent that only counts milliseconds is playing a different game from the person it is talking to.

How semantic turn detection works

The mechanism replaces the question "how long was the silence" with "do the words sound finished," as Zylos put it in July 2026. Streaming speech recognition emits partial transcripts continuously, and a lightweight classifier scores the current utterance for completeness at each update. "My order number is four seven" scores as incomplete. "My order number is four seven two nine one" scores as complete. Both may be followed by the same three hundred milliseconds of silence.

Production systems fuse that signal with the acoustic one rather than replacing it. Audio contributes falling pitch, final lengthening, and energy decay; text contributes syntactic and pragmatic closure. The combination sets an adaptive threshold: when linguistic completeness is high, respond almost immediately; when it is low, extend the silence window and keep listening. The result is a system that is fast on finished sentences and patient on unfinished ones, which is the behavior a fixed threshold cannot express at any setting.

The latency budget

Endpointing is one term in a chain, and it is the one teams control most directly. After the decision to respond come model inference, dominated by time to first token, then speech synthesis, then network and playback. Every component is additive, and the customer perceives only the total.

This is why the difference between waiting out a fixed second of silence and deciding in roughly three hundred milliseconds matters more than it looks. It is not a small percentage of the budget; it is often the largest single term, and it is spent before any useful work has begun. Shaving it frees room for everything downstream, which is what lets an AI voice agent land inside the range where a caller stops noticing the delay and starts treating the exchange as a conversation.

Where semantic turn detection still fails

Linguistic completeness is a better signal than silence, not a perfect one.

Trailing off. Callers abandon sentences constantly. "I wanted to ask about the, um" never becomes complete, so a purely semantic detector waits indefinitely. A silence ceiling has to remain as a backstop.

Disfluencies and list reading. Filled pauses, restarts, and self-corrections make transcripts look unfinished when the speaker is mid-thought and finished when they are not. Reading a list of items produces a sequence of complete-sounding fragments, each one an invitation to interrupt.

Language structure. Zylos notes the approach degrades on languages with frequent internal clause pauses, giving Chinese as an example, where a natural mid-sentence break resembles a completed thought to a model tuned on English. A detector validated in one language cannot be assumed to transfer, and word error rate on the upstream transcript sets a hard ceiling on how well the semantic layer can perform at all.

How it interacts with barge-in and backchannels

Turn detection decides when the agent may start. The mirror problem is deciding what to do when the customer speaks while the agent is already talking, and the two share machinery.

Not every incoming sound is an interruption. "Mm-hm," "right," and "okay" are backchannels, listener signals that mean keep going, and an agent that stops for each one becomes impossible to talk to. A real barge-in is different in content and usually in duration and prosody. Callsphere names barge-in confidence as the companion signal for telling the two apart, scoring incoming speech for whether it is a genuine bid for the floor before the agent yields.

Both decisions run on the same fused audio and text stream, and both benefit from the same instrumentation. Logging endpointing latency, false interruptions, and missed barge-ins per conversation turns a subjective complaint about an agent feeling awkward into three numbers a team can move.