Time to first token (TTFT)

Time to first token (TTFT) is the latency between sending a request to a model and receiving the first output token. It measures how long the system takes to start answering, not how long it takes to finish.

For streamed text, TTFT is most of what a user experiences as speed, because everything after the first token arrives while they are already reading. For voice, it is stricter than that: TTFT plus time to first audio determines whether the agent begins speaking inside the pause a caller expects or leaves a silence that reads as a dropped call. It is the single number that decides whether a conversation feels responsive or dead on the line.

What TTFT measures

TTFT is a wall-clock interval with three components, and only one of them is inference. Network transit carries the request to the provider and the first token back. Queueing is time spent waiting for capacity before the request is scheduled at all. Prefill is the forward pass over the entire input prompt, which must complete before any output token can be produced.

Distinguishing them matters because they respond to different interventions. Queueing is a capacity and provider problem. Network transit is a region and connection problem. Prefill is a prompt problem, and it is usually the term a team can move furthest with its own decisions. Measuring TTFT as one opaque number hides which of the three is actually hurting, which is how teams end up shortening prompts to fix what was really a queueing problem.

What happens during prefill

Before a model can predict anything, it processes every token of the input to build the internal state the first output token is generated from. Prefill is parallel across input positions and compute-bound, which is why it is fast per token and still the dominant term when the input is large.

The consequence is that TTFT scales with input length. Double the prompt and prefill roughly doubles. This links TTFT directly to context discipline: a long conversation history, an oversized retrieval payload, and a wall of tool schemas are not only quality and cost problems, they are latency problems the customer feels on every turn. MCP context bloat is the clearest case, because tool definitions are fixed overhead paid in full on the first turn and every turn after, and the same growth that causes context rot also lengthens prefill. Trimming the prompt improves accuracy and speed together, which is rare enough to be worth exploiting.

TTFT versus inter-token latency versus total latency

Three latency numbers get conflated, and they behave differently.

Time to first token. How long until output starts. Dominated by prefill and queueing, and therefore by input size.

Inter-token latency. How fast tokens arrive after the first, sometimes reported as output tokens per second. This is memory-bandwidth-bound decoding, and it is essentially independent of prompt length.

Total completion time. TTFT plus inter-token latency times output length. This is the only one that matters for non-streamed workloads, and the least useful for interactive ones.

The distinction is practical. A system with excellent throughput and poor TTFT feels sluggish in chat even though its aggregate numbers look good, and reporting median inference time without separating these terms will hide it. Perceived responsiveness in a streaming interface is mostly TTFT; perceived fluency once speech is underway is mostly inter-token latency.

Why voice agents are the strictest consumer of TTFT

Text tolerates a pause. A user watching a cursor blink for a moment reads it as thinking. Voice does not, because silence on a phone line means something specific and unhelpful to a caller.

An AI voice agent also spends its budget before the model is even reached. Turn detection has to decide the caller is finished, and after the model responds, speech synthesis has to produce audible audio and the network has to deliver it. TTFT sits in the middle of that chain, and whatever it consumes is unavailable to everything else. This is why voice deployments often route the first hop differently from the rest of the turn, optimizing purely for how quickly speech can begin rather than for the quality of the complete response.

How teams reduce TTFT

The levers fall into two groups: make the input smaller, or make the path to the first token shorter.

Shorten the prompt. The most reliable improvement, because prefill scales with input. Retrieve narrowly rather than stuffing, compact the history, and scope tool definitions per use case.

Cache what repeats. Prompt caching lets a provider reuse computed state for an unchanged prefix, cutting prefill on the repeated portion, which rewards putting stable instructions first and volatile content last. Semantic caching goes further for high-frequency questions by serving a stored answer without inference at all.

Shorten the first hop. A smaller or distilled model can produce an opening response while a larger one handles the substance, and speculative decoding uses a draft model to reduce the latency of early tokens. Both trade some quality or complexity for a faster start.

Buy capacity. Provisioned throughput removes queueing variance, which is often the difference between a good median and an unacceptable tail.

One caution: throughput optimizations frequently trade TTFT away. Aggressive request batching raises tokens per second per GPU by making individual requests wait, and batch inference abandons interactive latency entirely by design. Those are the right choices for offline work and the wrong ones for a conversation, so conversational and analytical traffic should not share a serving configuration.