Batch inference

Batch inference is the practice of processing many LLM requests together as an asynchronous job rather than one at a time in real time. The user submits a batch of prompts, the model provider queues and processes them in bulk over minutes to hours, and the results are returned when the batch completes. In exchange for tolerating latency measured in hours instead of seconds, users typically pay 50% or less of the real-time inference cost.

Batch inference is a core primitive in the modern LLM stack for any workload that doesn't need immediate responses. Data enrichment, offline analysis, evaluations, content generation pipelines, and knowledge-base rebuilds are all natural fits. All major LLM providers now offer batch APIs, and the discount they provide meaningfully changes the economics of large-scale LLM usage.

How batch inference works

Batch APIs share a similar structure across providers. The user submits a batch as a file (typically JSONL, with one request per line) containing many independent prompts. The provider accepts the batch and returns a batch ID. The batch is processed asynchronously — typically with a completion window of a few hours to 24 hours. When the batch completes, the user retrieves the results as another JSONL file.

The provider's incentive to offer a batch discount comes from operational efficiency. Real-time inference requires GPU capacity kept warm and idle for peak demand. Batch inference can run during off-peak hours, on lower-priority hardware, or across GPUs that would otherwise sit underutilized. The provider passes some of that efficiency back to the user as a discount.

When batch inference is the right tool

Batch inference is a strong fit for four categories of workload.

Data enrichment. Classifying, tagging, or summarizing large corpora — millions of support tickets, documents, or product listings — where the output feeds a downstream analytical system rather than a user waiting for a response.

Evals and testing. Running an eval suite of thousands of examples on a new prompt or model version. Evals typically don't need to complete in seconds; they need to complete overnight so the team can review results in the morning.

Content generation pipelines. Producing long-form content, product descriptions, or personalized outreach at scale where the pipeline runs on a schedule.

Knowledge base preparation. Chunking, embedding, or entity-extracting large document sets when building or refreshing a knowledge base. The one-time indexing pass for a large corpus is dramatically cheaper with batch inference.

Trade-offs

The main trade-off is latency. A batch that takes six hours to complete is unusable for anything a user is waiting on. Even for asynchronous workloads, unpredictable batch completion times complicate downstream scheduling — a batch expected to finish in two hours but taking eight hours can miss a downstream dependency.

The second trade-off is throughput uncertainty. Providers offer a completion window (often 24 hours) but don't guarantee faster completion. During periods of heavy demand, batches take longer to complete. Workloads with hard deadlines must either accept the worst-case completion time or fall back to real-time inference.

The third trade-off is limited feature parity. Some real-time API features may not be available in batch mode: streaming (obviously), some fine-tuned models, certain tool calling patterns, or certain model versions. Providers publish batch-mode compatibility separately from their real-time API.

Cost implications

Batch discounts vary by provider and model but typically fall in the 40-50% range compared to real-time API pricing. On top of this, batch APIs often have separate, higher rate limits than real-time APIs — a batch job of a million requests may be trivial through the batch API but would exhaust real-time rate limits many times over.

For workloads with high total inference volume and any latency tolerance, batch inference is one of the largest available cost optimizations. A team spending $100K per month on real-time inference for offline analytical workloads that could tolerate hourly completion could realistically cut that to $50K by moving to batch, without any change to the underlying model or prompt.

Batch inference in the modern AI stack

Well-designed AI systems distinguish clearly between real-time and batch workloads and route each to the appropriate infrastructure. Real-time paths — chat responses, live agent tool calls, interactive product features — need low latency and go through the real-time API. Batch paths — nightly evaluations, weekly analytical enrichment, one-time knowledge-base rebuilds — go through the batch API.

The distinction becomes especially important as inference volume grows. A team that runs everything through the real-time API pays the real-time premium even for workloads that don't need it. A team that routes appropriately treats real-time as a scarce, high-priority resource and batch as the default for anything that can wait. This kind of architectural clarity typically pays back within the first month of adoption.

For AI teams optimizing inference costs, batch inference sits alongside other techniques — context caching, model routing, prompt compression, and fine-tuning smaller models — in the toolkit for making high-volume AI economics work. Each has different applicability; batch inference is unique in that it applies to nearly every large asynchronous workload with almost no engineering cost to adopt.