Audio codec

An audio codec is an algorithm that compresses raw audio into a smaller data stream for transmission or storage, then decompresses it for playback. The word is a portmanteau of "coder-decoder." In voice AI and telephony, the codec determines how many kilobits per second your audio consumes on the wire, how much CPU is needed to encode and decode it, and how much of the original signal quality survives the round trip.

For an AI voice agent, codec choice sits between the caller's phone and the speech recognition pipeline. A poor codec choice can add tens of milliseconds of encoding latency, drop critical high-frequency content that the ASR model relies on, or increase packet loss sensitivity to the point where a dropped 20-millisecond frame becomes an audible glitch and a mis-transcribed word.

How audio codecs work

Codecs fall into two broad families. Waveform codecs like G.711 preserve the raw audio signal with minimal transformation — G.711 samples at 8 kHz and applies simple logarithmic compression, yielding 64 kbps of uncompressed-sounding audio. Parametric codecs like Opus and G.729 model the audio as a series of speech-generation parameters (pitch, formant frequencies, noise components) and transmit those parameters instead of raw samples, achieving much lower bitrates at the cost of computational complexity.

Every codec makes trade-offs across four dimensions: bitrate (how much bandwidth the stream consumes), sample rate (how much of the audible frequency range is captured), algorithmic latency (how many milliseconds of audio the encoder must buffer before it can produce output), and CPU cost (how much processing power encoding and decoding require). Real-time voice systems care about all four, but latency and packet-loss robustness usually dominate.

Common codecs in voice AI stacks

G.711 is the incumbent of traditional telephony. It transmits 8 kHz audio at 64 kbps using either μ-law (North America and Japan) or A-law (most of the rest of the world). Encoding is trivially fast and lossless-sounding within its 4 kHz bandwidth. The 8 kHz sample rate is called "narrowband" — high enough for speech intelligibility but low enough to make sibilants and some phonemes noticeably muted.

G.722 extends the sample rate to 16 kHz ("wideband") at 64 kbps, capturing frequencies up to 7 kHz. This makes s, f, and th sounds clearer and improves speech recognition accuracy for models trained on wideband audio.

Opus is the modern default for internet voice. It scales from 6 to 510 kbps and from 8 kHz to 48 kHz sample rate within a single codec, adjusting dynamically based on network conditions. Opus at 16 kHz sample rate and 24 kbps sounds better than G.711 at 64 kbps while using less than half the bandwidth. It has 20-millisecond default frame size and built-in forward error correction, making it resilient to packet loss.

G.729 is a legacy parametric codec still common on VoIP trunks. It runs at 8 kbps but introduces about 15 ms of algorithmic delay and produces audio that is intelligible but visibly degraded compared to G.711 or Opus.

Why codec choice matters for AI voice agents

An AI voice agent's response time is the sum of many small delays: capture, encoding, network transmission, decoding, ASR inference, LLM inference, TTS generation, encoding again, and playback. Codec choices affect two of those segments. Modern voice AI infrastructure prefers Opus end-to-end where possible because its low bitrate reduces jitter risk (see VoIP jitter) and its wider sample rate improves ASR accuracy.

The other reason codec matters is transcoding. When a call originates on a G.711 PSTN trunk but the AI processing runs in a cloud that expects 16 kHz Opus, the audio must be resampled and re-encoded at least twice per direction. Each transcode adds a few milliseconds of latency and destroys a small amount of signal fidelity. Voice AI teams that care about latency budgets increasingly negotiate with their telephony providers to expose native Opus streams or higher-quality G.722 rather than accepting G.711 by default.

Practical implications

Three codec-adjacent decisions repeatedly surface in production voice AI systems. First, resist the temptation to pick the lowest-bandwidth codec available — every kilobit saved on the wire costs latency and ASR accuracy in exchange, and voice bandwidth is a solved problem in most modern networks. Second, avoid unnecessary transcoding hops in your call path; a single call touching four codecs on its way to the AI system will have measurably worse MOS and WER than a call staying on one codec. Third, monitor codec negotiation logs during outages: many "voice AI is broken today" incidents turn out to be a carrier renegotiating calls onto a lower-quality codec due to congestion, and the observable failure is downstream ASR degradation rather than a codec alarm.