Chain-of-thought prompting

Chain-of-thought prompting is a technique that instructs a large language model to work through a problem in a sequence of intermediate steps before producing a final answer, rather than jumping straight to a conclusion. The simplest version adds an instruction like "think step by step" to the prompt. More advanced versions supply worked examples that demonstrate the reasoning pattern, or structure the reasoning process explicitly. On math, logic, and other multi-step reasoning tasks, the technique can improve accuracy by a wide margin compared to prompting for a direct answer.

The technique matters because it costs nothing beyond a longer prompt and a few extra output tokens, yet it can meaningfully change how reliably a model handles compositional problems. It became one of the most studied prompting techniques in the LLM literature starting around 2022, and it remains a standard tool for getting more reliable output from any model, not just the newest ones.

It is also the conceptual predecessor to the purpose-built reasoning models that now do this kind of deliberation natively, rather than needing to be prompted into it.

How chain-of-thought prompting works

Without prompting for reasoning, a model asked a compound arithmetic question will sometimes state a wrong number immediately, because the single forward pass that produces the first token of the answer does not have room to work through the intermediate arithmetic. With chain-of-thought prompting, the model is asked to reason first: it states the intermediate calculation, then the next one, and only then commits to the final number.

The mechanism behind the improvement is that the intermediate tokens give the model computational room to work through a problem incrementally rather than having to arrive at the correct answer in one step.

Each generated token can condition on everything generated before it, so writing out the reasoning effectively lets the model use its own prior output as scratch space.

Common variants

Zero-shot prompting. Adding a simple instruction such as "explain your reasoning before answering" to the prompt, with no worked examples required.

Few-shot prompting. Including a handful of worked examples that show reasoning followed by an answer, which the model imitates on the new problem.

Self-consistency. Sampling several independent reasoning chains for the same question and taking the most common final answer, which reduces variance at the cost of multiple model calls.

Tree-structured reasoning. Exploring multiple reasoning branches in parallel, evaluating each, and selecting the strongest one rather than committing to a single linear chain.

Program-aided reasoning. Letting the model reason by writing code and then executing that code to obtain the final answer, which is useful when the underlying problem is arithmetic or algorithmic.

Chain-of-thought prompting versus reasoning models

Chain-of-thought prompting is a prompt-side technique that works on any model and is fully controlled by whoever writes the prompt. Reasoning models, by contrast, are trained end-to-end to reason, so the chain of thought happens natively without needing to be prompted for it. See reasoning model for how that training-time distinction works.

Because their training explicitly rewards good reasoning, reasoning models tend to produce longer and more reliable chains than a standard model that has simply been asked to think step by step. Chain-of-thought prompting still earns its place, though, on older or cheaper models where dedicated reasoning training is not available, and as a lightweight technique that doesn't require switching model providers.

Trade-offs and failure modes

The technique is unnecessary overhead for simple retrieval tasks — a question like an order-status lookup gains nothing from step-by-step deliberation, and every reasoning token adds latency and cost. In latency-sensitive, synchronous conversations this overhead is often the deciding factor against using it at all.

Smaller models present a subtler failure mode: they sometimes produce reasoning that looks plausible but is actually a post-hoc rationalization rather than the real computation behind the answer, so the stated steps don't reliably reflect how the model actually arrived at its conclusion. Larger models benefit more consistently from the technique. There is also a hallucination risk: a confident, well-structured chain of reasoning can make an incorrect final answer more persuasive than a terse wrong answer would have been, so chain-of-thought output still needs to be paired with independent checks rather than trusted on the strength of its explanation. See hallucination detection for how production systems address this.

Where chain-of-thought prompting fits in an AI stack

Production AI agents typically apply chain-of-thought prompting selectively rather than everywhere. A system prompt might enable step-by-step reasoning for complex intents like disputes or policy exceptions, while leaving it off for routine intents like checking an order status, which keeps latency and cost under control without sacrificing accuracy where it matters. When paired with a RAG pipeline, the technique lets a model reason across multiple retrieved passages explicitly — this policy applies, the customer's account shows this state, therefore the answer is this — which is a large part of what makes an answer feel grounded rather than guessed.

End users are rarely shown the reasoning itself; most support-facing agents hide it entirely and surface only the final answer, since customers want the resolution rather than the model's internal deliberation. What they experience instead is the downstream effect: a customer support AI that gets multi-step, policy-dependent questions right more consistently because the reasoning happened before the answer was written, not as an afterthought.