Voice agent simulation

Voice agent simulation is the practice of testing a voice agent by having synthetic callers place real audio calls against it, across a range of accents, background conditions, emotional states, and interruption patterns, and scoring what happens on every turn before any customer hears the system. The calls are synthetic; the audio path is not. The agent under test receives sound, transcribes it, decides, and speaks back exactly as it would in production.

The discipline exists because voice agents fail in places text testing cannot reach. VAmoS Bench, published on arXiv in July 2026, introduced a voice agent simulation benchmark measuring complete voice-agent systems end to end on a stateful customer-support task, with roughly one hundred scenarios of which about a third apply adversarial pressure. Future AGI reported in May 2026 that simulating thousands of synthetic personas placing real audio calls compressed a six-week pre-launch test sprint to roughly three days.

Why voice agents need audio-level testing

A voice agent is a pipeline, and most of that pipeline does not exist in a text harness. Testing the language model with typed input validates one stage and assumes the rest are perfect.

Transcription is the first thing text testing skips. Real callers produce accented speech, disfluencies, crosstalk, and proper nouns the recognizer has never seen, and automatic speech recognition turns all of that into text with a nonzero error rate. An agent that reasons flawlessly over clean text may reason confidently over a mangled order number. Only an audio test exposes that interaction.

Turn-taking is the second. Whether the agent speaks at the right moment depends on voice activity detection and endpointing decisions that have no text equivalent. A caller who pauses mid-sentence to check a receipt will be cut off by an agent whose endpointing is too aggressive, and no transcript-based test will show it. The same is true of barge-in behavior when a caller interrupts a long response.

Latency and prosody complete the list. Response latency that reads as acceptable in chat is conspicuous dead air on a phone call, and pacing and intonation change how an answer lands even when the words are identical. Neither has a representation in text.

How a simulation harness works

Persona and scenario generation. The harness defines a caller: their goal, account state, temperament, cooperativeness, and the facts they will volunteer versus withhold. Scenarios are drawn from real contact-mix data where possible, so the distribution of intents resembles production.

Speech synthesis for the caller side. Each persona's turns are rendered to audio with a voice matching its profile, using speech synthesis with controlled prosody so a frustrated caller sounds frustrated rather than merely saying frustrated words.

A real audio path. The call is placed over the same transport production uses, so codec artifacts, packet loss, and jitter are part of the test rather than abstracted away. This is what separates simulation from a scripted unit test.

A driving model on the caller side. The synthetic caller is not reading a fixed script. A model listens to the agent's responses and decides what the persona says next, which lets the simulation explore branches, push back on wrong answers, and reproduce the messiness of a real multi-turn call. Every call is captured with audio, both transcripts, timing data, and internal tool calls.

What dimensions get varied

The value of simulation comes from coverage, meaning deliberate variation across conditions a small human test panel will never reproduce. Accent and dialect are the most obvious axis, because recognition quality is uneven across them. Acoustic conditions come next: street noise, a car, a speakerphone, a poor mobile connection. Emotional state is a third axis spanning calm, confused, rushed, and angry, and it changes both the acoustics and the behavior, since an angry caller interrupts more and volunteers less.

Interaction patterns form the fourth axis and are where most regressions hide. Callers who interrupt, trail off, change their mind halfway through, give a number one digit at a time, or speak over the greeting all exercise turn-taking logic a cooperative caller never touches. Adversarial scenarios belong here too: callers who try to talk the agent out of policy, supply contradictory identity information, or push for an action outside authorization.

What gets scored on every turn

Per-turn scoring is what makes simulation diagnostic rather than merely pass or fail. Recognition accuracy is measured with word error rate against the known caller-side text, a luxury simulation provides and production does not, since the harness knows exactly what the synthetic caller said.

Timing is scored per turn as well: how long the agent waited before responding, whether it cut the caller off, whether it left dead air, and whether it handled an interruption gracefully. These are the metrics semantic turn detection work aims to improve, and simulation is how the improvement gets measured.

Above the audio layer, scoring is the same as for any agent: was the response grounded, did it follow policy, did it call the right tools with the right arguments, did it escalate when it should have. Because the harness captures the internal trace alongside the audio, a failed call can be attributed to a layer, separating "the voice agent got it wrong" from "recognition dropped the account number and everything downstream followed correctly from bad input."

Where simulation fits alongside evals and production monitoring

Simulation is pre-release testing. It occupies the same position for an AI voice agent that a staging environment occupies for a web service: a place to exercise the whole system under controlled variation before real traffic arrives. It complements rather than replaces text-based evals, which remain the faster way to iterate on prompts, policies, and tool logic. A workable division of labor puts high-volume, low-cost evals on the reasoning layer and reserves simulation for audio-dependent properties and release regression suites.

After launch, the same scoring rubric moves to sampled live calls. Production monitoring is the only source of conditions nobody thought to simulate, and every genuine failure should end by becoming a new scenario in the harness. That loop, from harness testing before release to sampled scoring after it and back again, is what keeps a voice agent from degrading quietly once real callers find its edges.