Audio-native semantic speaker-change detection

The problem: knowing when another user joins the conversation

In customer support, voice agents need to know not only what was said, but who is participating. If another person takes over a call or begins instructing the customer, the agent may need to pause, re-authenticate, or restrict sensitive actions. Missing that change creates security and fraud risk, while flagging every background voice would constantly interrupt legitimate conversations.

Standard ASR-to-LLM-to-TTS pipelines cannot reliably make this decision. They miss out on whether speech came from the customer, another participant, a television, an announcement, or background voices. To solve this problem, we need dynamic audio-native solutions that can act on the raw waveform.

Benchmark construction

Production benchmarks should measure whether the system behaves correctly, not just whether a second voice is acoustically present. A loud airport announcement should not interrupt the call; someone taking over or coaching the customer through a sensitive action should. We therefore annotate only relevant speaker changes—those that should alter the agent’s behavior—and stratify ambiguous misses and false positives by their likely impact on conversation flow.

Because one false trigger can disrupt an otherwise successful call, we measure conversation-level false-positive rate (cFPR): the share of conversations with any false trigger before or without a relevant speaker change. Turn-level FPR understates this compounding risk:

For positive conversations, conversation-level recall asks whether the system triggers on the first request containing the relevant new speaker; a late trigger both misses the change and may fire after that speaker has left. We therefore maximize conversation-level recall at a fixed cFPR ceiling.

Solutions

Diarization

Diarization asks “who spoke when?” Models like Pyannote and Sortformer can be adapted by flagging whenever their output contains more than one speaker ID. But they are optimized to segment every acoustically distinct speaker, whereas our detector should trigger only when a change warrants intervention. A person speaking on a television is therefore a correct diarization event but a harmful false trigger for the agent.

This objective mismatch creates false triggers even when the acoustic segmentation is correct. Separately, these models also have high FPR on literal speaker-change detection, even on clean synthetic data; turn-level errors then compound at conversation level on production calls.

Speaker embeddings

Unlike a diarizer, speaker embeddings like TitaNet offer lower-level building blocks: it maps variable-length audio to an embedding space rather than directly identifying speakers or deciding who spoke when. It’s trained with an additive angular-margin (AAM) loss function that optimizes the embedding space for cosine comparison, so utterances from the same speaker should point in similar directions.

To turn embeddings into a speaker-change score, we split audio into overlapping windows and compare them with cosine similarity. We evaluated three methodologies:

These methodologies are cheap, fast, and outperform off-the-shelf diarization. They can be tuned for high recall, but their scores capture acoustic separability rather than relevance, so precision remains insufficient for a final decision. That makes them useful as an acoustic gate: we discard low-scoring candidates, then let a multimodal model judge whether the remaining changes are relevant.

Multimodal post-training

Multimodal LLMs combine audio comprehension with reasoning and holistic understanding. These models do not perform well on this task out-of-the-box, and prompting and in-context learning mostly move them along the cFPR-recall Pareto frontier, not past it. To improve performance, we focus on post-training these models.

Synthetic dataset generation

Post-training data must cover the conditions that drive production errors: conversation length and turn structure, background noise, and speaker variation. We considered three sources, each trading off acoustic realism, semantic coherence, distribution control, and scale:

• Human-recorded conversations provide the most natural voices and interaction patterns, but recording and review are slow and expensive. They are also difficult to balance across production conditions—especially less common combinations of age, accent, pitch, prosody, and timbre.
• Spliced speech corpora, such as LibriSpeech and VCTK, preserve real voices and make it easy to control turn counts, durations, and noise at scale. But their utterances were typically read in isolation, so the resulting exchanges lack conversational and semantic coherence.
• LLM-generated conversations rendered with TTS provide coherent scenarios, precise semantic control, and scalable coverage of the production distribution. Their weakness is acoustic fidelity: cloned voices do not reproduce the full variation in human prosody, timbre, and other fine-grained vocal cues.

In our SFT experiments evaluated on production conversations, adding training data improved AUC: the model ranked relevant changes above irrelevant ones more reliably overall. But higher AUC did not necessarily improve the low-cFPR tail, where a small number of difficult false positives determines production performance.

Supervised fine-tuning from first principles

At inference, the model produces a yes/no answer, but its token log-probabilities provide a continuous score. Thresholding the yes/no log-odds lets us choose the recall–cFPR operating point.

This normalization ignores any residual probability assigned to other tokens; with a constrained yes/no response format, that mass is typically negligible.

Taking the negative log of the conversation-level probability gives:

Under this factorization, the conversation-level negative log-likelihood decomposes into binary cross-entropy on the normalized yes/no probabilities. This is effectively standard maximum-likelihood SFT. The gradients make the equivalence explicit:

These gradients push every specificity request toward no and the single recall request toward yes using ordinary cross-entropy. Paired with log-odds thresholding, SFT directly optimizes the scores used at deployment. On production conversations, SFT with a TitaNet cohesion gate achieved three times the recall of diarization models at 1% cFPR.

DPO is another option, but it underperformed SFT in our experiments. DPO optimizes a relative preference margin rather than the calibrated yes/no likelihood used at a fixed deployment threshold. Binary cross-entropy is better aligned with this objective: its updates scale with probability error, concentrating the largest corrections on confident mistakes.

Future of audio models

Many general-purpose audio-language models still use representations and objectives optimized for transcription or response generation, which can discard evidence needed for acoustic reasoning. This limitation is not universal: TitaNet, WavLM, CLAP, and BEATs are explicitly trained to preserve speaker or broader acoustic information.

However, there are still several major gaps in today’s audio models. Semantic tokens are compact but often remove fine acoustic detail; codec tokens preserve more of the waveform but produce much longer sequences. Preserving the signal is only half the problem: models must also be trained to use evidence across timescales, from timbre and intonation to rhythm and harmony.

What is missing is an audio-native analogue of InstructGPT: information-preserving representations, a broad curriculum of audio-native tasks, and evaluations that require answers to be grounded in the waveform. Models such as Qwen3-Omni move in this direction, but broad, reliable audio reasoning remains an open problem.