RAG pipeline

A RAG pipeline is the end-to-end system that connects a language model to an external knowledge base so that its responses are grounded in that knowledge rather than in the model's parametric memory alone. Where retrieval-augmented generation describes the general technique, the pipeline is the concrete implementation: the ingest, embedding, storage, retrieval, and generation stages wired together into a running system.

Nearly every production AI application that answers questions from proprietary content runs some form of RAG pipeline. Customer support agents, internal knowledge assistants, documentation search, and legal or medical research tools all share this basic architecture. The pipeline is what turns a static knowledge base into a live, queryable interface — and its quality determines whether an AI system feels helpful or hallucinatory.

The five stages of a RAG pipeline

A RAG pipeline has five stages that split cleanly into an offline ingestion path and an online query path.

Chunking. Source documents are split into smaller passages, typically 200 to 1,000 tokens each. Chunk size affects retrieval quality: too small and passages lack context, too large and retrieval becomes imprecise. Most production pipelines chunk with some overlap between adjacent passages so information that spans a chunk boundary is not lost.

Embedding. Each chunk is passed through an embedding model that converts the text into a high-dimensional vector representing its semantic content. Chunks with similar meaning produce vectors close to each other in the embedding space, which is what makes semantic retrieval possible.

Storage. The vectors and their source chunks are written to a vector database or a hybrid vector-plus-keyword index. This index is what the pipeline queries against at runtime. For large corpora, indexing is expensive and typically runs as a batch job when the underlying content changes.

Retrieval. At query time, the user's question is embedded through the same model, and the vector database returns the top-k chunks whose embeddings are closest to the query embedding. Well-designed pipelines apply reranking, filtering, or hybrid keyword-plus-vector search here to improve relevance beyond raw embedding similarity.

Generation. The retrieved chunks are inserted into the LLM's prompt as context, along with the original user question and a system prompt describing the task. The LLM synthesizes an answer that draws on the retrieved passages, ideally citing them. This is the only stage a user ever sees directly.

What makes a RAG pipeline good

The gap between a demo RAG pipeline and a production RAG pipeline is enormous, and almost all of that gap lives in retrieval quality. A model can only answer well from context it actually receives, so a pipeline that retrieves the wrong chunks will produce confidently wrong answers even with a state-of-the-art LLM.

The highest-leverage improvements typically involve better chunking strategies, hybrid retrieval that combines semantic and keyword search, query rewriting so the retrieval query differs from the raw user question when that helps, reranking with a more expensive model over the top-k initial results, and metadata filtering to constrain retrieval to a relevant subset of the index. Each of these can meaningfully raise the fraction of queries where the pipeline retrieves passages that actually answer the question.

Evaluation matters equally. A serious RAG pipeline is evaluated with an eval suite that measures retrieval precision, retrieval recall, answer faithfulness, and answer helpfulness on a fixed set of representative queries. Without evals, pipeline changes are guesses; with evals, changes are measured improvements.

Where RAG pipelines break

Even a well-tuned pipeline has failure modes worth naming, because they show up predictably in production and dictate what to build next.

The first is the multi-document synthesis problem. Vector retrieval returns a handful of passages, but some questions require aggregating information across dozens of them. "What are all the products that integrate with Salesforce?" cannot be answered from three retrieved chunks even if all three chunks are relevant. Systems that face this pattern often move toward Graph RAG or precomputed summaries to give the LLM broader coverage.

The second is the stale-index problem. Vector indexes reflect the state of the underlying corpus at the last indexing run. When a policy changes, a price updates, or a product is discontinued, the pipeline continues to return the outdated chunk until the index is rebuilt. Production pipelines need explicit re-indexing schedules, change-detection hooks, or streaming ingestion to keep freshness within acceptable bounds.

The third is the ambiguity problem. When a user asks a vague question, retrieval returns technically-similar but off-target chunks, and the model synthesizes an answer that sounds authoritative but misses the user's actual intent. Query rewriting and clarification-first patterns help, but this is fundamentally a product design problem as much as a retrieval problem.

RAG pipeline versus fine-tuning

A common question when building on proprietary content is whether to use a RAG pipeline or fine-tune the model on that content instead. In practice, these are complements more than alternatives, and the choice depends on what property the team needs.

RAG pipelines are the right choice when the underlying content changes frequently, when responses need to cite specific source passages, when the corpus is large enough that fine-tuning would be expensive, or when different queries need to be answered from different subsets of the content. Fine-tuning is the right choice when the model needs to internalize a style, format, or domain-specific reasoning pattern rather than specific facts.

Serious production systems often use both: a fine-tuned model that knows the domain's terminology and preferred response format, running on top of a RAG pipeline that supplies the up-to-date factual content it needs for any given query.

Where RAG pipelines fit in an AI stack

A RAG pipeline sits at the center of nearly every AI application that operates over proprietary content. For a customer support AI agent, the RAG pipeline is what lets the agent reference the actual help center rather than making up plausible-sounding answers. For an internal enterprise assistant, it is what lets the assistant answer questions from company documents without leaking that content into a general-purpose model.

The pipeline is rarely visible to end users, but it is almost always the single largest determinant of whether the AI system feels trustworthy. Investment in the pipeline — better chunking, better retrieval, better evals, faster re-indexing — pays back in every user interaction. For AI teams building on top of proprietary knowledge, the RAG pipeline is the foundation everything else depends on.