Teaching flow-matching text-to-speech models with RL

Good demos, unreliable distributions

Modern text-to-speech (TTS) systems can sound remarkably natural. But average quality hides the failures that matter most: a rushed sentence, misplaced pause, or flattened intonation can dominate a listener's impression of an otherwise convincing interaction.

The goal of post-training TTS models is not to just achieve a better average sample, but a tighter distribution: fewer tail failures, no leakage onto ordinary prompts, and no loss of intelligibility, speaker identity, or naturalness.

Supervised fine-tuning is the foundation. It teaches a model what a behavior looks like and is useful when that behavior is absent. But once the model can produce several plausible generations, the remaining problem is comparative. Which generation has the right pace, pause, intonation, or expressive event?

We study two settings: control tags—inline cues such as [cough], [pause], and [slow] that request specific behaviors—and naturalness optimization.

Choosing the right signal, and why flow models complicate it

The right training method depends on where the learning signal comes from:

‍

Applying DPO and GRPO to modern speech models requires resolving a basic mismatch. Flow-matching models generate speech by learning how to continuously transform noise into audio. Once the initial noise is chosen, generation follows a deterministic trajectory rather than a sequence of sampled actions with explicit probabilities.

A token model samples discrete outputs from an explicit distribution, so each choice has a log-probability. A flow-matching model instead transforms noise into audio through a learned velocity field:

An autoregressive backbone conditions a flow head that generates the next latent audio patch. Each transition is a deterministic update, not distribution-sampled action. That leaves DPO without a policy-versus-reference score and GRPO without old and current action log-probabilities.

Our first workaround was Reward-rAnked Fine-Tuning (RAFT) which avoids log-probabilities by sampling several takes, ranking them by reward, and fine-tuning on the winners. It improved cough reliability, but the result did not generalize cleanly across controls. Pause RAFT either over-paused or traded firing for consistency, while prosody remained near the supervised fine-tuning ceiling.

That limitation motivated Flow-DPO and Flow-GRPO.

Flow-DPO: preferences without exact audio likelihoods

We replace the unavailable policy-versus-reference log-probability gain with a negative relative flow loss[2]:

Flow loss is not an exact waveform likelihood, and it only provides the relative signal DPO needs to separate preferred and rejected trajectories.

That separation depends on clean pairs. In our first Flow-DPO experiment for the [cough] control tag, many rejected samples still contained real coughs, so voice and intelligibility differences carried the preference signal instead of the event itself. We rebuilt the dataset around same-prompt contrasts that isolated the cough, then filtered preferred samples for intelligibility, speaker consistency, and artifacts.

Flow-GRPO: creating probabilities inside a deterministic flow

GRPO converts rewards for several rollouts from the same prompt into group-relative advantages. Token models apply those advantages through old and current action log-probabilities, but deterministic flows have no equivalent transition probability.

Flow-GRPO[3] converts the deterministic ODE into a stochastic differential equation (SDE) by adding Gaussian noise at selected steps, giving each transition a probability. But noise alone would change the model's distribution. To correct for that change, we estimate the score from the velocity field:

Training only the flow head is a cheap way to validate the method, but it limits how far the policy can move. We unfreeze the autoregressive backbone only after teacher-forced conditioning matches rollout conditioning, with fixed-reference KL and held-out evaluation as guardrails. In our runs, the two conditioning paths reached a cosine similarity of about 0.998, clearing the alignment check before full-model training.

Reward design: targets, shortcuts, and constraints

A control-tag reward has three jobs: produce the requested behavior, suppress it when unrequested, and preserve the surrounding speech.

Pause control showed how easily those goals can conflict. Because our reward penalized unwanted pauses and word errors more heavily than missed requested pauses, the policy learned to pause less everywhere. Mean reward improved by 35% even as requested-pause success fell from roughly 50% to 29.2%. The optimizer was working as intended; the objective was not.

The same pattern can appear across speech controls:

• An event reward can increase requested activations while also increasing leakage onto ordinary prompts or changing the speaker's voice.
• A pacing reward can be satisfied through silence or warbling rather than slower articulation.
• A naturalness reward can be over-optimized until the voice drifts from the reference.

To prevent these shortcuts, we separate the target behavior from the properties that must be preserved:

• Negative controls cover prompts where the behavior should not occur.
• Preservation gates reject unintelligible, off-speaker, or artifact-heavy samples regardless of reward.
• Reference anchors limit broad drift from the starting policy.
• Two-sided targets optimize toward a validated range rather than maximizing a proxy without bound.
• Held-out evaluation tracks component metrics and blind listening, not only the training reward.

Naturalness was harder because no single metric captured it reliably. Rollout consistency worked well for finding unusually bad takes and constructing DPO pairs, but optimizing agreement with the model's own outputs risked rewarding bland, mode-seeking speech. Our most stable Flow-GRPO runs instead used an externally grounded reward[7] combining speaker similarity, ASR accuracy, and DNSMOS[16] quality.

Results: the method follows the feedback

Across coughs, pauses, and pace, the same methods behaved differently depending on how well the feedback captured the target behavior and its preservation constraints.

‍

Naturalness proved harder to optimize. Flow-GRPO with the speaker-similarity, ASR, and DNSMOS reward produced more consistent high-quality generations and eliminated the tail failures seen in the baseline.

Across these experiments, two patterns stood out:

First, Flow-DPO is robust when clean contrasts exist. Pair purity and clean-prompt controls made the conditional behavior explicit. On coughs, Flow-DPO delivered strong, stable performance across target and preservation metrics, while GRPO over-optimized the reward, producing overly aggressive events, increased leakage, and lower speaker similarity.

Second, Flow-GRPO enables larger policy movement. That helped on pace, where the reward directly measured articulation rate and intelligibility was gated. The difference was reward quality: stronger optimization helped when the reward matched the intended behavior and hurt when it did not.

From speech models to voice agents

The goal extends beyond better TTS: it is a voice agent whose words, timing, prosody, and nonverbal behavior work together as a coherent policy. At Decagon, we're pursuing this by optimizing language and acoustics to improve naturalness without sacrificing reliability.

References

Flow matching and continuous-model alignment

• [1] RAFT — Reward rAnked FineTuning for Generative Foundation Model Alignment. arxiv.org/abs/2304.06767
• [2] ARDM-DPO — Direct Preference Optimization for Speech Autoregressive Diffusion Models. arxiv.org/abs/2509.18928
• [3] Flow-GRPO. arxiv.org/abs/2505.05470
• [4] MAR-GRPO — GRPO for masked-autoregressive models with diffusion heads. arxiv.org/abs/2604.06966 · Code
• [5] Diffusion-DPO. arxiv.org/abs/2311.12908
• [6] DanceGRPO. arxiv.org/abs/2505.07818 · Code

TTS and audio post-training

• [7] Qwen 3.0. arxiv.org/abs/2607.23938
• [8] FlowTTS-GRPO. arxiv.org/abs/2606.23190
• [9] F5R-TTS. arxiv.org/abs/2504.02407
• [10] DMOSpeech 2. arxiv.org/abs/2507.14988 · Code
• [11] Seed-TTS. arxiv.org/abs/2406.02430

Reward design and evaluation

• [12] No Verifiable Reward for Prosody. arxiv.org/abs/2509.18531
• [13] SpeechJudge. arxiv.org/abs/2511.07931
• [14] Attacking UTMOS. arxiv.org/abs/2606.31105
• [15] Align2Speak. arxiv.org/abs/2509.21718
• [16] DNSMOS. arxiv.org/abs/2010.15258