Retrieval-Augmented Generation (RAG)

What RAG Is: An Open-Book Exam for LLMs

Retrieval-Augmented Generation (RAG) is an architecture that retrieves external knowledge first, then has the large language model (LLM) generate its answer based on what was retrieved — turning the model from "reciting from memory" into "taking an open-book exam."

Imagine asking a knowledgeable consultant whose knowledge stops last year about your company's newest policy: he will either make something up or say "I don't know." RAG's approach: hand him a constantly updated manual first, and restrict him to "answer only from the manual; if it isn't in there, say you don't know." That is the entire spirit of RAG.

It has been the most popular LLM application paradigm since 2023 because without any training or parameter changes, "retrieve + assemble a prompt" alone can turn an LLM from a generalist into a trustworthy specialist in a domain. Customer service bots, enterprise knowledge base Q&A, AI search, legal and medical assistance all lean on it heavily. As of 2025, nearly every mainstream LLM product ships with some form of retrieval built in. This page first clarifies the problem RAG solves, then dissects the architecture, maps the variants, compares the alternatives, and closes with the pitfalls people hit most often in production. For the conceptual big picture and the arc of model evolution, start with What Are the Hot AI Concepts and Large Language Models (LLM).

1. Why You Need RAG: Three Birth Defects of LLMs

A large language model is at bottom a parameterized probability distribution — it "compresses" its training corpus into hundreds of billions of parameters. That mechanism comes with three unavoidable weaknesses:

Weakness	Symptom	Consequence
Hallucination	Confidently fabricating things it doesn't know; the more confident the wording, the harder to spot	Factual errors; unusable in enterprise settings
Knowledge cutoff	Training data has a cutoff date; the world after it is invisible	Ask about the latest policy or today's events and it will get them wrong
No access to private data	Training corpora exclude internal company documents and personal files	Clueless about your company knowledge base and proprietary data

Hallucination is rooted in this: at generation time the model has no ability to "look things up" — it relies on the memory in its parameters, and that memory is fuzzy statistical regularity, not reliable fact. For a deeper analysis of the hallucination mechanism, see Large Language Models (LLM).

RAG is the architecture-level answer to all three weaknesses. Its value boils down to three points:

1. Traceable (Grounding): answers rest on specific retrieved text and can carry citations; users can verify them, and errors can be traced to a source.
2. Updatable (Fresh): updating knowledge = replacing documents in the index — no retraining. Swap documents today, answers change tomorrow.
3. Cheap (Cost-controlled): no training, no parameter tuning — just one extra retrieval at inference time. Compared with repeated fine-tuning, the marginal cost is tiny, which makes it ideal when "knowledge changes often but model capability is good enough."

Rule of thumb: when the model's capability falls short, consider switching models or fine-tuning; when what's lacking is knowledge that isn't fresh, complete, or private enough, reach for RAG first.

2. Architecture: A Three-Stage Pipeline

RAG is a clean pipeline with three stages: offline indexing, online retrieval, and generation. The overall architecture:

┌─────────── Offline indexing (built once; rerun when documents change) ──────────┐
│  Raw documents → clean → split into chunks → embed → write to the vector store  │
└────────────────────────────────────────────────────────────────────────┘
                          │ (reused at answer time)
┌─────────── Online retrieval (runs on every question) ────────────┐
│  User question → Query embedding → vector recall (ANN)            │
│                        → rerank → hybrid retrieval                │
└────────────────────────────────────────────────────────────────────────┘
                          │ (top-k relevant chunks)
┌─────────── Generation ──────────────────────────────────────┐
│  Retrieved chunks + user question → assemble prompt → LLM answers (with citations) │
└────────────────────────────────────────────────────────────────────────┘

For the complete implementation, see Build a RAG App from Scratch; here we dissect each stage's principles and traps.

1. Offline Indexing: Squeeze the Knowledge into Storage First

The offline stage turns raw documents into a searchable vector index, in three key steps:

Step 1: Chunking. Cut long documents into right-sized "knowledge blocks." This is the single biggest factor in RAG success or failure, yet it is often handled carelessly.

• Common strategies: split by fixed character count, by paragraph/heading, by semantic boundary (e.g. a sentence splitter), or by token limit;
• The core tension: chunks too small → lost context, fragmented retrieval; chunks too large → diluted vector semantics, irrelevant information mixed in, wasted LLM context window;
• Common traps: tables sliced through the middle, code blocks cut off, cross-chunk semantics broken, technical terms split apart.

Three rules of chunking

1. Split by document structure first, then by size: Markdown headings and PDF sections are natural semantic boundaries;
2. Start at 200–500 tokens per chunk, and experiment with real questions — don't just guess;
3. Use one chunking strategy per document type: PDFs, web pages, and codebases each have their own best practices; mixed corpora need separate handling.

Step 2: Embedding. Use an embedding model to map each chunk into a dense vector (e.g. 1536/1024 dimensions); semantically similar texts land close together in vector space. This step is the foundation of Vector Databases and Semantic Search.

# Pseudocode: the offline indexing flow
from embeddings import EmbeddingModel
from vector_db import VectorStore

docs = load_documents()                    # 1. Load documents
chunks = split_by_structure(docs, size=300) # 2. Split by structure, target ~300 tokens
vectors = [EmbeddingModel.embed(c) for c in chunks]
store = VectorStore(dimension=1024)
store.upsert(ids, vectors, metadata=chunks) # 3. Write to the vector store along with metadata

Step 3: Record metadata. Every chunk must carry metadata — source, title, timestamp, permission labels, and so on. They matter more than most people expect: they power post-hoc filtering, source attribution in answers, and permission-scoped retrieval. RAG without metadata means "you have knowledge, but you don't know where it came from or whether you're allowed to use it."

2. Online Retrieval: Fishing the Most Relevant Pieces Out of the Ocean

Retrieval quality caps answer quality — if retrieval can't find the right answer, no LLM is smart enough to save you. Online retrieval typically stacks three techniques:

• Vector retrieval (semantic search): embed the user's question and run a nearest-neighbor (ANN) search in the vector store, taking the top-50. Strong on queries that are "semantically close but worded differently";
• Reranking: a cross-encoder model scores and reorders the top-50, keeping the top-5. Vector retrieval is the "first pass," reranking is the "fine pass" — this is the highest-leverage step for RAG accuracy per unit of cost, though note that it adds one model inference and increases latency;
• Hybrid retrieval: vector search plus keyword search (e.g. BM25), with the two rankings fused by an algorithm like RRF (Reciprocal Rank Fusion). For "literal match" tasks — proper nouns, model numbers, IDs — BM25 is often more accurate than vector search, so the two complement rather than exclude each other.

Rule of thumb: in production, start with "vector + BM25 hybrid + reranking" — the best accuracy-for-cost balance; if you need extreme low latency, cut reranking first.

3. Generation: Fencing the Answer Inside the Retrieved Material

Assemble the top-k retrieved chunks with the user's question into a prompt and hand it to the LLM. The core discipline here is stating explicitly in the prompt "answer only from the provided material" — a classic application of Prompt Engineering. A reusable template:

You are an enterprise knowledge base assistant.
Answer the user's question strictly from the [Reference material] below;
if the reference material contains no answer, reply "No relevant
information in the knowledge base" — never invent anything.
Tag every conclusion with its [source number].

[Reference material]
[1] Source: FAQ/refund-process.md — Refunds are typically returned via the
    original payment method within 3–5 business days; during major sale
    events this may stretch to 7 business days. ... (retrieved chunk goes here)
[2] Source: after-sales-policy/2025.md — ... (second chunk goes here)

[Question]
How long does a refund take to arrive after I request one?

[Answer]

Why does "answer only from the material" work? The retrieved results "pin" the model's attention onto the given text, sharply compressing the space for hallucination; combined with the "say you don't know" instruction, it explicitly rules out the uncertain cases. But note: this is only a soft constraint — the model can still wander beyond the material, so production systems also stack harder measures on top, such as citation tagging and source verification (see Common Pitfalls and Anti-Patterns).

3. The RAG Variant Family: From Bare Bones to Fully Armed

RAG is not a single solution but a rapidly evolving family. Under the mainstream survey's classification (see References), it falls into three generations:

Variant	Core idea	Typical techniques	Best for
Naive RAG	Split → retrieve → stuff the prompt; works out of the box	The simplest three-stage pipeline	Validating ideas, quick launches
Advanced RAG	Optimize both ends of retrieval	Query rewriting/expansion, HyDE, reranking, chunk tuning, metadata filtering	Production-grade accuracy requirements
Modular RAG	Every retrieval stage is pluggable and orchestratable	Query routing, multi-way recall, memory modules, retriever fusion	Complex business logic, many data sources
GraphRAG	Retrieve using knowledge graph structure	Entity/relation retrieval, community summaries, global Q&A	Cross-document reasoning, summarization-type questions
Self-RAG	The model itself decides "when to retrieve, what to retrieve, whether to accept it"	On-demand retrieval, critical acceptance, cited output	Cost-sensitive tasks where retrieval can hurt quality
Agentic RAG	An agent autonomously plans "when to look, what to look up, how many times"	Multi-step retrieval, tool calls, self-correction, reflection	Complex multi-hop Q&A, workflow-type tasks

A few key judgments:

• Most teams ship while still stuck at Naive RAG, then blame "RAG doesn't work" on what is really worse retrieval quality. Get the four workhorses of Advanced RAG in place first (query rewriting, reranking, chunk tuning, metadata filtering) before debating anything else;
• GraphRAG excels at "summarization-type" tasks (e.g. "what issues do all our product lines share?") because it organizes knowledge into an entity-relation network rather than fragments — a fit for teams with existing knowledge graph assets or a need for a global view. See Knowledge Graphs and Knowledge Injection;
• Agentic RAG suits "multi-hop" questions ("compare policies A and B, then assess the impact on C") because it can retrieve → judge → retrieve again; the price is higher latency and not-fully-predictable behavior. See AI Agents;
• Self-RAG is the more "on-demand" take: it hands the "should I retrieve?" decision to the model itself (with gating learned via special tokens during training) — simple questions get answered directly, retrieval happens only for complex ones, and the model critically evaluates what it cites. It saves tokens and improves precision, but requires a specially fine-tuned model behind it. See Frontier Research.

4. RAG vs. Fine-Tuning vs. Long Context: How to Choose

When "the model doesn't know something," the industry has three mainstream fixes: RAG, fine-tuning, and stuffing everything into a long context. They aren't substitutes — they are different choices on different cost curves:

Dimension	RAG	Fine-tuning	Long context
Knowledge updates	Swap documents; effective immediately	Requires retraining / continual updates	Just edit the prompt
Traceability	Strong (can carry citations)	None (knowledge lives in parameters)	Weak (manual checking only)
Hallucination risk	Low-to-medium (bounded by retrieval quality)	Low (but it will "remember" errors)	Medium-high (key details get drowned out)
Cost	Storage + retrieval + inference	Training + inference (the most expensive)	Grows linearly with context length
Latency	Low–medium	Low	Medium–high (long inputs are slow)
Best at	Factual Q&A, knowledge bases, time-sensitive info	Style/format/domain adaptation, capability injection	Deep single-document analysis, long-form reading

How to choose? Three rules of thumb:

1. Knowledge lives in documents → RAG. Customer service, enterprise knowledge bases, policies and regulations, product docs — the answers are "ready-made" in your materials; RAG is the default first choice;
2. Knowledge belongs in parameters → fine-tuning. When you need to change how the model talks, its output format, or its domain vocabulary — or want it to "answer right without looking anything up," like learning your API calling conventions. See Fine-Tuning and PEFT (LoRA);
3. One-off long-document understanding → long context. Summarizing or answering questions over a 100K-word report? Feed it straight in — don't build an index first.

Combos are the norm

Real systems combine all three: RAG backs up the facts, fine-tuning backs up the style, long context handles one-off large documents. A customer service bot = fine-tuned "support tone" + RAG over the policy library + long context to read the contract the user just uploaded.

5. Production Essentials: Four Things That Get Underestimated

RAG looks simple in a demo; four things get underestimated on the road to production:

1. Retrieval quality evaluation (unmeasured retrieval is flying blind). Don't just watch "how good are the answers" — quantify "how accurate is retrieval" first.

• Metrics: recall (is the right answer in the top-k?), hit rate, MRR (ranking quality);
• End to end: use an evaluation framework like RAGAS to measure answer faithfulness (does the answer stick to the material?), answer relevancy (is it on topic?), and context precision/recall (how good is the retrieved context?) — see LLM Evaluation and Benchmarks;
• Practice: assemble 50–200 real question-answer pairs as an eval set, and watch the metrics move as you iterate on retrieval parameters — never tune by feel.

2. Chunk-size tuning (experimentation is the only reliable method). Freeze everything else, vary only the chunk size, run the eval suite, and watch the hit-rate curve. A common pattern: as chunk size climbs from 100 to 500, hit rate first rises, then falls — too small loses context; too large dilutes semantics. There is a sweet spot in between.

3. Vector store selection (decide by scale and ops capacity). From "embedded lightweight libraries" to "distributed heavyweights," it's a gradient:

Tier	Typical representatives	Scale	Characteristics
Embedded	Chroma, FAISS, SQLite-vec	Tens of thousands of chunks	Zero ops; fast start
Open source servers	Milvus, Qdrant, Weaviate	Millions	Scalable; needs ops
Cloud hosted	Pinecone, cloud-vendor vector stores	Any scale	Fully managed; pay as you go
PostgreSQL-based	pgvector	Hundreds of thousands	Same stack as your business DB; transactional consistency

For selection details and how semantic search works, see Vector Databases and Semantic Search. Don't agonize over the choice at the start — get running on an embedded library and migrate when scale demands it.

4. Cost and latency (do the math up front). RAG's hidden costs are easy to ignore: embedding generation and storage fees, the extra inference cost of reranking, QPS pressure on the vector store, and the maintenance cost of an index that grows with your documents. Optimization directions: result caching (cache hits for identical questions), tiered retrieval (cheap recall first, fine reranking only on the top-k), vector quantization, and general-purpose inference optimizations — see Inference Optimization and Quantization.

6. Real Products: RAG Is Already Everywhere

RAG is not a lab toy — top products long ago made it a core capability:

• Perplexity: the benchmark AI search engine. User asks → live web crawling → retrieval → cited answer generation, with a source list at the bottom of every answer for item-by-item verification. It made "traceability" the core selling point of the product experience — see the full teardown in Perplexity and AI Search;
• Notion AI (business tier): lets users "Ask AI" directly over workspace documents, meeting notes, and databases, with every answer linking back to the source page — textbook enterprise knowledge base RAG;
• Microsoft Copilot (Bing mode): web retrieval + generation — RAG in the browser;
• Customer service / legal / medical assistants of every stripe: retrieve the relevant policy clause or statute, answer, and attach the clause number.

These products share one formula: a good RAG product = 60% retrieval quality + 30% interaction design + 10% generation. To bring Perplexity home, work through Build a RAG App from Scratch yourself.

7. Limitations and Common Misconceptions

RAG is an engineering solution, not a silver bullet; it has its own boundaries:

Limitation 1: if retrieval fails, generation fails. RAG's answer quality is capped by retrieval quality — when the right answer isn't recalled, the LLM may confidently fabricate something that "sounds plausible." Retrieval is RAG's Achilles' heel, which is why "needle in a haystack" tests (burying a key fact in a very long document) keep producing failures: it's not that the model can't, it's that the retriever can't find the needle.

Limitation 2: evaluation is hard. Generative answers have no single right answer, and "was this response good" depends on human judgment; blame for retrieval quality, generation quality, and prompt effectiveness gets tangled together. For building an evaluation system, see LLM Evaluation and Benchmarks and Building an LLM Eval System.

Limitation 3: the context window is a finite resource. Too many retrieved chunks crowd the window and dilute attention; too few may miss key evidence. It takes careful budgeting.

Three high-frequency misconceptions

1. "RAG = prompt assembly": concatenate without optimizing retrieval and the results will be poor; retrieval optimization is the main battlefield;
2. "Smaller chunks are better": small chunks improve pinpointing but lose context — answers are often "correct but incomplete";
3. "With RAG you don't need fine-tuning": RAG handles "knowledge"; fine-tuning handles "capability and style." They solve different problems.

For a more systematic catalog of failures and how to avoid them, see Common Pitfalls and Anti-Patterns.

Further Reading

• Large Language Models (LLM) — the generation foundation RAG rides on, and how hallucination arises
• Vector Databases and Semantic Search — the core technology of RAG's retrieval layer
• Prompt Engineering — designing the "answer only from the material" prompt
• Knowledge Graphs and Knowledge Injection — the structure beneath GraphRAG
• AI Agents — Agentic RAG and multi-step retrieval
• Fine-Tuning and PEFT (LoRA) — RAG's alternative and complement
• Inference Optimization and Quantization — keeping RAG latency and cost under control
• LLM Evaluation and Benchmarks — quantifying retrieval and generation quality
• Perplexity and AI Search — the flagship RAG product case study
• Build a RAG App from Scratch — the hands-on guide

References

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020) — the original RAG paper; introduced the joint "retriever + generator" training paradigm
• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey (2023) — the RAG survey; source of the Naive/Advanced/Modular classification
• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023) — the end-to-end RAG evaluation framework and its metrics
• Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024) — Microsoft's GraphRAG paper
• Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009) — the classic BM25 keyword-retrieval literature
• LlamaIndex documentation — one of the de facto standard RAG frameworks, full of chunking/retrieval best practices