RAG Explained: How Retrieval-Augmented Generation Works
From chunking and vector search to reranking and tenant isolation: how retrieval-augmented generation actually works, grounded in a real production system.
Ask ChatGPT about your company’s refund policy and it will answer confidently — and wrongly. It has never seen the policy. RAG is the fix: fetch the right paragraph first, then let the model write. In interviews, though, nobody asks ‘what is RAG?’ They ask ‘your bot gave a wrong answer — was it retrieval or generation?’ By the end of this guide you’ll answer that in under two minutes.
Explain it like I’m five
Imagine a helper who is great at talking but cannot remember every fact. Before answering, the helper opens a bookshelf, finds the best page, reads it, and then explains what it says.
That is RAG: retrieve the most useful pieces of information, then let the model generate an answer from them. If the bookshelf returns the wrong page, the answer may still sound confident—so finding good evidence is just as important as writing well.
Intuition: an open-book exam
Think of it like an open-book exam: the student never memorizes the textbook — they just need to find the right page fast, then write the answer from it. A practical RAG system has two paths. Offline, it splits documents into chunks, converts each chunk to an embedding, and stores those vectors in an index. At query time, it embeds the question, retrieves nearby chunks, optionally reranks them, and places the best evidence into the model’s context window.
Retrieval happens before generation. The model receives both the question and selected evidence; it does not “look inside” the index itself.
Retrieval finds candidate evidence; generation writes an answer. If the right fact is not retrieved, a perfect generator still cannot ground its answer in that fact.
The mechanism, one piece at a time
1. Chunk the source
Documents are too large and semantically mixed to retrieve whole. Split them into units that preserve meaning: often a heading plus several paragraphs. Tiny chunks lose context; giant chunks dilute similarity and waste the prompt budget.
2. Embed and index
An embedding model maps text to a fixed-length vector. Related meanings land near one another. A vector index uses approximate nearest-neighbor search to find likely matches quickly, trading a small amount of recall for large speed gains.
3. Retrieve, filter, rerank
Semantic similarity is a good first pass, not a final answer. Production systems mix dense retrieval with keyword/BM25 search, apply metadata filters, then use a stronger reranker on the shortlist.
4. Generate with explicit evidence
The prompt should separate instructions, evidence, and question. Ask the model to abstain when evidence is insufficient and to attach citations that can be checked against chunk identifiers.
DoorDash: support knowledge that learns from conversations
DoorDash documented a production support chatbot whose approved knowledge-base articles are surfaced through a RAG layer, so the bot can answer from new operational knowledge without retraining the language model.
The problem was scale: manually maintaining support content could not keep pace with the issues appearing in customer conversations. DoorDash’s system turns recurring issue patterns into proposed articles, routes them through approval, embeds the approved material in a vector database, and makes it available to the chatbot.
Approved support articles become searchable embeddings. The knowledge can change independently of the model, which matters when policies and edge cases evolve.
At conversation time, the system compares the live issue with stored issue embeddings and retrieves the matching article instead of relying only on what the model learned during training.
The chatbot combines the retrieved article with the current conversation and context, then writes the answer. Retrieval supplies the evidence; the language model supplies the explanation.
In DoorDash’s reported online A/B test, escalation in high-traffic issue clusters fell from 78% in control to 43% in treatment. About 75% of treatment retrieval events used only the newly created user-generated content.
Source: DoorDash Engineering, Aug. 18, 2025 ↗The new knowledge lives in approved articles and is fetched at answer time. The model’s weights do not need to change each time the support library does.
A worked numeric example
Suppose the query embedding is q = [1, 1]. We normalize vectors and use cosine similarity.
A top-1 retriever chooses A. If A were split so “30 days” lived in a different chunk, retrieval could return a semantically relevant but incomplete fragment. That is a chunking failure—not an LLM failure.
Build the smallest version
This deliberately omits embedding API calls and the generation model. It isolates the retrieval contract you should be able to explain and test.
import numpy as np
def cosine_scores(query, documents):
# rows of documents are precomputed chunk embeddings
query = query / np.linalg.norm(query)
docs = documents / np.linalg.norm(documents, axis=1, keepdims=True)
return docs @ query
def retrieve(query_vec, chunk_vecs, chunks, k=3):
scores = cosine_scores(query_vec, chunk_vecs)
top = np.argsort(scores)[-k:][::-1]
return [(chunks[i], float(scores[i])) for i in top]
def build_prompt(question, evidence):
context = "\n\n".join(text for text, _ in evidence)
return f"Answer only from the evidence. Cite chunk ids.\n{context}\nQ: {question}"In production, add chunk IDs, source URLs, access-control metadata, an approximate index, hybrid retrieval, and telemetry for every stage.
What interviewers actually ask
These are realistic practice prompts shaped around common company interview emphases—not claims about a private question bank.
What to say (≈90 sec): “I’d split it into three checks. First, was the right evidence retrieved? I look at recall@k against a labeled eval set and inspect the retrieved chunks for the failing queries. Second, given the right evidence, did the model use it? I run an oracle test — feed the gold chunk directly into the generator. If the answer is still wrong, it’s a generation problem, not retrieval. Third, was the answer faithful? I check that every factual claim is attributable to a cited chunk, not to the model’s prior. I track these separately: retrieval recall, faithfulness rate, and end-to-end correctness — because a single accuracy number can’t tell you which stage broke.”
Likely follow-up: “You don’t have labeled data. Now what?” → Bootstrap synthetic question–chunk pairs with a strong LLM judge, then hand-verify a sample — a hundred clean pairs beat ten thousand noisy ones, and they give you a recall@k baseline you can trust.
The answer that sinks you: “I’d try a bigger model.” That’s a fix without a diagnosis — if retrieval was the problem, the bigger model gets the same bad evidence.
What to say (≈90 sec): “I treat every retrieved document as untrusted data, never as instructions. First, I enforce a hard structural separation: system and user instructions are privileged, while retrieved text goes into a clearly delimited evidence block the model is trained to treat as data. Second, I preserve instruction hierarchy, so document text can’t override user or system intent. Third, I sanitize: strip control tokens and prompt-delimiter phrases, normalize whitespace, and drop content matching known injection patterns. Fourth, I constrain tools — least-privilege scopes, no direct tool invocation from retrieved content, and confirmation for sensitive actions. Finally, I evaluate: a red-team suite of injection attempts with a measured attack-success rate, rerun on every prompt change.”
Likely follow-up: “The injection is a polite paragraph, not an obvious ‘ignore your instructions’ — what catches that?” → Behavioral checks: flag any tool call or recommendation the user never asked for, compare agent actions against user intent in a shadow eval, and send anomalous actions to review. The signal is what the agent does, not the wording of the poisoned text.
The answer that sinks you: “I’d just tell the model in the system prompt to ignore malicious documents.” A bare instruction is the weakest layer — cleverer phrasing overrides it. You need structural separation, hierarchy, and tool gating, not a wish in the prompt.
What to say (≈90 sec): “I’d design around four problems: scale, freshness, deletion, and measurement. For scale, I shard by document hash across machines, each holding an approximate index — HNSW or IVF — then fan the query out and merge top-k with each shard’s distance scores. For freshness, I run async indexing: a streaming pipeline that embeds new versions and appends them while the old version keeps serving. For deletion and updates, I use tombstones so a deleted document stops being returned before the index segment is rebuilt, and I version every embedding. For correctness under filters, I apply metadata filters before the ANN search or over-fetch and filter after — never filter a shortlist down to nothing. And I measure throughout: recall@k against a sample of exact search, p99 latency per shard, and indexing lag as a first-class metric.”
Likely follow-up: “A document updates and a user queries five seconds later. What do they see?” → The new version: the pipeline writes the new embedding with a higher version, the tombstone suppresses the old one, and I track the lag between document commit and index visibility as an SLO. If lag exceeds the SLO, that’s an alerting signal, not a silent miss.
The answer that sinks you: “One big exact index, recomputed nightly.” Exact search at a billion documents is infeasible per query, and nightly rebuilds make freshness a joke. You need approximate search plus a streaming update path.
What to say (≈90 sec): “Hybrid wins when the query carries exact tokens that embeddings smear. Dense retrieval is great for semantic paraphrase — ‘refund timeframe’ matching ‘return window’ — but it compresses rare, specific strings: product SKUs, error codes like ERR_742, version strings, proper names, and domain terms with one precise spelling. BM25 nails those because it matches the literal token. So I run both, fuse the ranked lists — reciprocal rank fusion or a tuned weighted score — and rerank the shortlist with a cross-encoder. On a corpus of technical docs where queries mention part numbers, hybrid reliably beats dense-only on recall@k, and the reranker cleans up the fused list.”
Likely follow-up: “How do you set the fusion weights?” → Tune them on a labeled eval set optimizing recall@k or nDCG — start at an even split, grid-search, and if query types segment cleanly, learn per-segment weights. Revisit the weights whenever the corpus or query mix shifts.
The answer that sinks you: “Dense embeddings capture meaning, so keyword search is obsolete.” Embeddings can blur the one token that matters — a support bot that can’t match the literal error code in the ticket fails the exact queries users ask most.
What to say (≈90 sec): “Isolation has to be enforced before retrieval, not after. Every chunk carries authorization metadata — tenant id and role claims — bound to the index entry at write time. The tenant’s identity goes into the retrieval query as a mandatory pre-filter pushed down into the vector search, so another tenant’s vectors are never scored, never ranked, never logged in a result set. I avoid post-hoc filtering entirely: it leaks through score patterns, timing, and logs. On top of that, every retrieval is audited with the requesting tenant’s identity, embeddings are encrypted per policy, and I run isolation tests — query as tenant B and assert zero tenant-A content can surface, including via reranking or caching layers.”
Likely follow-up: “A document is shared with a second tenant, then unshared. What has to happen?” → The ACL metadata on every chunk of that document must change atomically with the sharing decision — update the index entries or tombstone and re-embed before the permission change takes effect, then run the isolation test as the revoked tenant to prove it’s gone. Stale ACLs are a breach.
The answer that sinks you: “Retrieve first, then filter out anything the tenant shouldn’t see.” Post-retrieval filtering leaks — cross-tenant data still flows through scoring, caches, and logs. Filter before retrieval, at the index.
What to say (≈90 sec): “The whole design gets squeezed into a memory, power, and privacy envelope. I switch to a small distilled embedding model and quantize vectors — int8 or product quantization — so the index fits in a few hundred megabytes. The corpus is tiered: a compact high-value index on device, synced incrementally with deltas rather than full rebuilds. At query time I use a small ANN probe count, cache hot queries, and budget the compute so a search doesn’t drain the battery. Telemetry stays privacy-preserving — aggregated or differentially private, never raw queries off device. And I build graceful fallback: under memory pressure, shrink the probe count, serve from cache, or abstain rather than hang or crash.”
Likely follow-up: “Even quantized, the index won’t fit in RAM. Now what?” → Memory-map the index from flash and page in what’s needed, cut the corpus to the highest-value subset for that user, and accept slower cold queries. If the working set still won’t fit, fall back to server retrieval for the long tail and keep the hot set on device.
The answer that sinks you: “I’d run the same server index on the phone.” A phone doesn’t have the RAM, thermal budget, or battery for a server-grade index — you’d OOM on the first query. On-device means redesigning for the envelope, not porting the server.
Key takeaways
- RAG changes the model’s input, not its weights.
- Retrieval quality has its own metrics: recall@k, precision@k, MRR, and nDCG.
- Chunking, reranking, and access control matter as much as the vector database.
- Evaluate retrieval, faithfulness, answer quality, latency, and cost independently.
- Good RAG systems say “I don’t know” when evidence is missing.