Small language model

A small language model, or SLM, is a language model with roughly 1 to 8 billion parameters — small enough to run on a single GPU or even consumer hardware, cheap enough to serve at very high query volume, and increasingly capable on tasks that once required a much larger model. Open examples include Phi-4, Llama 3.1 8B, Mistral 7B, and Qwen 2.5 7B, alongside closed models like Claude Haiku and GPT-4o-mini that occupy a similar size and price tier without publishing exact parameter counts. The defining trait is not any single capability but the combination of low serving cost, low latency, and a parameter count small enough to run outside a large multi-GPU cluster.

As production AI systems scale to millions of daily queries, where a query is routed — to a frontier model or to an SLM — has become one of the largest levers on both cost and latency, which is why model size has turned into an architectural decision rather than a footnote.

Small versus large: where the line falls

There is no formal industry cutoff between small, mid-size, and large models, but rough conventions have settled out as usage has grown. A small model, or SLM, generally sits under roughly 8 billion parameters and runs on a single GPU, often on consumer-grade hardware.

Mid-size models fall in the 10 to 70 billion parameter range, which requires meaningfully more GPU memory but is usually still feasible on a single well-equipped machine.

Large or frontier models start around 100 billion parameters and often run into the trillions using mixture-of-experts architectures, and they are served almost exclusively by large cloud providers rather than run independently.

Why small models matter now

Cost and latency. An SLM can often serve a query for a small fraction of what a frontier model would charge for the same request, which matters enormously at the volumes production AI agents now handle. Smaller models also tend to produce a faster time to first token, often well under a second, compared to the multi-second delays common with the largest frontier models.

Because an SLM's weights and compute footprint are modest, it can run inside a private cloud environment, on-premises, or even on-device, which is a natural fit for teams with strict data-residency requirements. The same small footprint makes fine-tuning economically realistic per customer or per narrow use case, something that is rarely practical with a frontier-scale model.

What small models are good at

Classification and extraction. Tasks such as intent detection, sentiment analysis, ticket routing, and pulling structured fields out of an invoice or contract play to an SLM's strengths, since the decision space or output format is narrow and well-defined even when the input text varies.

Retrieval-grounded answers. When paired with a well-built RAG pipeline, an SLM can produce answers competitive with much larger models, because the retrieved context does most of the heavy lifting and the model mainly needs to synthesize it faithfully. Modern small models also handle structured output and function calling reasonably well, which makes them viable for triggering actions rather than only answering questions.

Their low cost per call additionally makes small models a natural choice for a second-pass reviewer — reranking retrieved passages or running a safety check on a larger model's output — where running an expensive frontier model twice on every request would be wasteful.

Trade-offs and where small models fall short

Long, open-ended reasoning is the clearest limitation. Multi-step math, complex planning, and genuinely novel problems still tend to favor frontier-scale models, or purpose-built reasoning models regardless of their size.

Context length is a second constraint: most small models cap out well below the context windows available on the largest frontier models, which limits how much retrieved material or conversation history they can consider at once.

Thin tails and safety. Small models also have thinner knowledge in the tails — they tend to know common facts and patterns well but perform noticeably worse on rare, obscure, or highly specialized domain knowledge than a much larger model trained on a broader corpus. Model size by itself says nothing about safety, either: a poorly aligned small model can behave less safely than a well-aligned frontier model, since safety comes from training data curation, alignment techniques, and runtime guardrails rather than from parameter count.

Where small language models fit in an AI stack

Most production AI-agent stacks do not choose between a small model and a frontier model — they use both, with an LLM router sending routine, well-scoped work to an SLM and reserving the frontier model for complex reasoning and multi-step planning. In this pattern, the small model typically handles triage, classification, and simple lookups, while also running safety checks and grounding evaluations as an inexpensive second pass over the frontier model's output. Layering in techniques like prompt caching compounds the savings further, since the base cost of an SLM call is already low.

This kind of routing is what makes it possible to run a high-volume customer support AI agent economically: most incoming questions are simple enough that a small model resolves them directly, and only the harder share of conversations — the ones that require real reasoning or unusual account states — escalate to a larger model or a human. Decagon's platform applies this pattern so that model cost scales with problem difficulty rather than with raw conversation volume.

Net effect. Well-scoped use of small models keeps a support system fast and affordable at scale, while reserving larger, more expensive models for the smaller share of conversations that genuinely need them.