Reasoning model

A reasoning model is a large language model trained to spend significant inference-time compute generating an internal chain of thought before producing its final answer. Rather than committing to a response in a single forward pass, the model works through the problem in a sequence of intermediate steps that are mostly hidden from the end user, then condenses that work into a shorter final output. OpenAI's o-series, Anthropic's Claude with extended thinking, Google's Gemini Thinking models, and DeepSeek R1 all fall into this category. The trade is speed for accuracy: reasoning models are slower and more expensive per query, but they perform substantially better on math, code, planning, and other multi-step problems.

Reasoning models emerged as a distinct model class starting in 2024, once labs found that reinforcement learning could reward a model for producing longer, self-correcting chains of thought rather than short, confident-sounding answers. The result is a model that behaves less like an instant responder and more like something working a problem out on paper before speaking.

As AI agents take on more consequential, multi-step work, the choice of when to invoke a reasoning model versus a standard model has become one of the more consequential architecture decisions in a production LLM stack.

What makes a model a reasoning model

Two things separate reasoning models from standard LLMs. The first is training: reasoning models are typically trained with reinforcement learning to produce long, structured chains of thought that try an approach, catch a mistake, backtrack, and self-correct before committing to an answer. Standard models are trained primarily to predict the next token well, not to deliberate.

The second is inference-time compute. At runtime, a reasoning model generates thousands to tens of thousands of intermediate tokens that the user never sees, then emits a much shorter final answer. This is the core mechanism: for problems where the correct answer depends on several careful steps, giving the model room to work through them internally produces better results than forcing an immediate response. The cost of that room is latency and token spend, since every hidden reasoning token still has to be generated and paid for.

Reasoning models versus prompted reasoning

It is worth distinguishing a reasoning model from a standard model that has simply been prompted to reason. A standard LLM answers in one pass and is fast and cheap, which works well for retrieval and simple generation tasks. A standard LLM given chain-of-thought prompting — an instruction like "think step by step" — often improves its accuracy on multi-step problems, but the underlying model was never specifically trained to reason; the improvement comes from the prompt, not the weights.

A true reasoning model is trained end-to-end to reason, so the chain of thought is native behavior rather than something coaxed out by a prompt. It uses substantially more inference-time compute by default.

Its intermediate reasoning also tends to be longer, more self-correcting, and more reliable than what a prompted standard model produces on the same problem, precisely because the training process rewarded that behavior directly rather than relying on an instruction to elicit it.

Where reasoning models help and where they don't

Good fits. Reasoning models earn their cost on problems with several dependent steps: solving a math or logic problem, debugging a program that touches multiple files, planning a multi-step agentic workflow, or applying a written policy across several conditions to reach the correct outcome. They also help on ambiguous cases where a standard model reliably guesses wrong because the surface-level pattern match points to the wrong answer.

Poor fits. They are the wrong tool for simple retrieval — looking up an order status does not benefit from deliberation — and for high-throughput classification tasks like intent detection, where the extra compute is pure overhead. In synchronous, real-time conversations, the added latency, which can run from several seconds to over a minute depending on problem difficulty, is often unacceptable to a person waiting for a reply. Cost compounds the same way: reasoning models routinely cost several times more per query than standard models, so applying them indiscriminately across a high-volume workload is expensive without a proportional gain in outcome quality.

Measuring reasoning quality

Benchmarks. Reasoning ability is typically evaluated on tasks with checkable answers — competition math, graduate-level science questions, and coding benchmarks. These public benchmarks are useful for comparing models in the abstract, but they don't always predict performance on a specific production workload, which is why teams typically build their own evals that reflect the actual distribution of tasks a deployed system will see.

Cost per correct answer. The more useful production metric is often cost per correct answer rather than accuracy alone. A reasoning model that is ten percentage points more accurate but twenty times more expensive and considerably slower may not be worth deploying broadly, even though it wins on a leaderboard.

Reasoning also does not uniformly reduce factual errors: it tends to help on problems with a checkable, derivable answer, but it does not manufacture knowledge the model never had, so a reasoning model can still produce confident, well-structured answers that are factually wrong. See hallucination detection for how production systems catch this independently of reasoning quality.

Where reasoning models fit in an AI stack

Most production systems do not run every query through a reasoning model. Instead, a cheap classifier or an LLM router decides which turns warrant the extra compute, sending the bulk of straightforward traffic to a fast standard model and reserving the reasoning model for cases that genuinely need multi-step deliberation.

Reasoning models also frequently serve as the planning component inside a multi-agent system, where one model plans a course of action and cheaper models execute the individual steps.

In a customer-support deployment, this selective pattern is the difference between a system that is both fast and accurate and one that is either too slow or too shallow. Most turns — order lookups, account questions, routine requests — are handled by fast, inexpensive models. A smaller share of turns, such as disputed charges, policy edge cases, or multi-part requests that require reconciling several pieces of account information, get routed to a reasoning model precisely because the cost of a wrong answer there is higher than the cost of the extra compute. Decagon's AI agents apply this kind of routing so that reasoning-heavy compute is spent only where it changes the outcome, keeping typical conversations fast while still handling the harder cases correctly.