Prompt caching

Prompt caching is a feature offered by LLM providers that stores the model's processing of a prompt's prefix — a system prompt, tool definitions, retrieved documents, or few-shot examples — so that later requests sharing the same prefix skip reprocessing it. Those requests pay only a fraction of the normal input-token cost. The provider hashes the marked prefix, stores the resulting internal state, and on a matching request jumps straight to processing whatever new content follows, typically the latest user message.

The technique is distinct from semantic caching, which caches entire responses rather than partial computation. Prompt caching matters because most production LLM applications repeat a large, static block of context on every single call.

A customer-support agent sends the same lengthy system prompt, the same tool schemas, and often the same reference documents thousands of times a day, with only the final user message changing. Without caching, the provider reprocesses that entire static block from scratch on every request. With caching, that cost collapses, which is why prompt caching became a standard line item in LLM cost optimization within a year of major providers shipping it.

How prompt caching works

A request sent with caching enabled marks a prefix — the system prompt plus tools plus retrieved context, for example — that the provider processes once and stores under a hash of the exact prefix bytes. Any later request with an identical prefix skips that reprocessing and starts from the cached state, appending only the new suffix.

Cache hits typically cost 10% to 25% of the normal input-token price and complete two to five times faster than an uncached request of the same size. Implementations differ across providers, though the underlying idea stays the same.

Anthropic exposes explicit cache breakpoints that a developer sets manually within a prompt. OpenAI applies caching automatically on supported models without any special markup. Google's Vertex AI offers explicit context caching as a distinct API surface. All three reuse computation on a byte-identical prefix and charge less for doing so.

What is worth caching

Long system prompts. A multi-thousand-token system prompt shared across every request is close to a textbook case — cache it once and every subsequent call avoids paying to reprocess it.

Tool definitions. Detailed tool schemas rarely change between calls and are prime candidates, since they are large, static, and repeated on nearly every turn of an agentic loop.

Retrieved context. In a RAG pipeline where the same document gets referenced repeatedly within a conversation, caching that document avoids reprocessing it on every follow-up question.

Prior conversation turns. In a multi-turn conversation, earlier turns naturally become the cached prefix for whatever comes later, so long-running conversations benefit more the longer they run.

Prompt caching versus semantic caching

The two techniques solve different problems and are frequently confused because both use the word cache. Prompt caching reuses computation on a shared prefix — the model still runs and generates a genuinely fresh response every time, just at lower cost and latency.

Semantic caching reuses an entire past response, skipping the model call completely when a new query is close enough in meaning to an old one. Production stacks commonly run both: semantic caching to eliminate redundant calls entirely, and prompt caching to make every remaining call cheaper.

Getting prompt caching right, and where it falls short

Ordering matters most. Caching only works on prefixes, so stable content — the system prompt, tool definitions — needs to sit at the front of the request, with dynamic content like the latest user message appended after it. Reordering a prompt so static content moves around between calls breaks the cache even if the content itself hasn't changed.

Cache lifetime is a real constraint. Providers evict cached prefixes after a period of inactivity, often just minutes, so the benefit concentrates in high-volume, tightly clustered traffic rather than sparse or bursty workloads.

There is also a write cost to account for. Anthropic, for example, charges roughly 1.25 times the normal input rate on the first write of a new prefix, which means a prefix used only once or twice actually costs more than not caching at all. Workloads with highly variable retrieved context on every call get little benefit, since the cached prefix rarely repeats exactly.

Where prompt caching fits in an AI stack

Customer-support AI agents are close to an ideal use case for prompt caching: a long shared system prompt, the same tool definitions on every turn, and frequent reference to a stable core knowledge base across thousands of concurrent conversations. Deployments that adopt prompt caching in that setting typically see substantial reductions in input-token cost.

They also see a faster time-to-first-token, which a customer experiences directly as a snappier, more responsive agent. Combined with semantic caching and routing through an AI gateway, prompt caching is one of the more mechanical but highest-leverage cost controls available once a support AI system is running at meaningful volume.