RAG works on one PDF. Production hands you eleven thousand.

The tutorial has a PDF and a question. You split the PDF into chunks of a thousand characters, embed them, drop them into a vector store, embed the question, take the top five by cosine, paste them into a prompt and ask the model to answer from context only. It works on the first run. The answer cites page four. You ship a demo on Friday and by Monday somebody has sold it.

The customer corpus arrives as a bucket. Eleven thousand documents, of which some unknowable fraction are the same policy saved by six departments, half the PDFs are scans with OCR artefacts, a third of the “documents” are email threads that quote each other in full, and the dates on the files are the upload dates rather than the dates anything happened. A thousand people will query it, most of them about things that are not in it. Nothing in the tutorial covered any of this, because the tutorial was a demonstration of the mechanism, not a description of the job. The job is what follows.

The corpus is not the tutorial’s corpus

The first production surprise is that retrieval quality is mostly a property of the data, and the data has a shape you did not choose. Chunk a real corpus and look at where the embeddings land. The tutorial corpus, one book, spreads its chunks across semantic space because a book says different things in different chapters. A corporate corpus does not. It says the same few things many times, in slightly different words, with different headers, and the embedding model faithfully maps all of those to nearly the same point.

That has a specific and brutal consequence. Your query lands near a dense cluster and the top ten are ten copies of one document, each chunk a few tokens different from the next. You asked for ten pieces of context and got one piece of context ten times, while the nine documents that would have added something sit at rank twelve through twenty, never making it into the prompt. The model then answers from a single source and sounds confident about it. Nothing errored. Recall on paper is fine, because one of the ten copies is the right document. The answer is worse than it would have been with a deduplicated corpus one tenth the size.

The objection is that deduplication is an afternoon of work: hash the text, drop the repeats, done. Exact duplicates are that easy, and you should do it on ingest. The production problem is the near-duplicate that is legitimately distinct. Version three of a policy and version four differ in one clause, and the one clause is the answer. The same contract template exists for two hundred customers with different names and amounts. Every page carries the same four hundred characters of legal footer, which after chunking becomes a chunk in its own right, embedded eleven thousand times. Near-duplicate detection with a similarity threshold will happily collapse the two policy versions and keep all eleven thousand footers, which is the opposite of what you want. There is no universal knob. You dedupe by document identity where it exists, strip boilerplate structurally before chunking, and keep true versions as versions with a date field, which we will come back to.

So the first thing to instrument is ingest, before anything about queries. How many chunks per document. How concentrated the embedding space is. How many chunks have a near-neighbour above 0.98 that comes from a different document id. Those numbers tell you whether the retrieval problem you are about to tune is even the one you think it is.

Three metrics, one golden set

The learning-stage version of this is thirty questions with known answers and a script that reports whether the right chunk lands in the top five. The production version keeps the idea and grows it in two directions: more kinds of query, and more than one stage measured per query. A RAG pipeline has at least three places to fail, and a single end-to-end score cannot tell you which one did.

The point of measuring all three against the same set of queries is localisation. When the end-to-end number drops after a deploy, the aligned metrics tell you whether the retriever stopped finding the document, the reranker started burying it, or the generator began ignoring context it was given. Each of those has a different fix and a different owner, and without the split you will spend a week changing the prompt to repair a regression in the chunker. The golden set is the harness for the retrieval system in exactly the sense I used in the previous post: a second system that measures the trajectory of the first, positioned at every boundary where an assumption meets reality.

The set has to hold more than answerable questions. It needs questions whose answer is in the corpus and should be found; questions whose answer is not in the corpus, where the correct response is to say so; questions that are on-topic but require a document the user’s identity should not see; questions that are off-domain entirely; questions that are only correct for a particular date range; and adversarial questions built to pull a fact out of a document that should not be reachable. For each, the expectation is a grounded fact whose presence or absence you check in the response, not a reference answer you fuzzy-match. “Contains the figure 14 days” is checkable. “Resembles this paragraph” is not.

The set rots, so treat it as living

The sharp objection is that a golden set for retrieval decays faster than the corpus it measures. Every incremental index update changes which chunk is the best evidence for a question, so a check pinned to a chunk id is wrong within a week, and the set drifts into a suite people mark as expected-failure and stop reading. This is real and it is why the checks are written against facts rather than chunks. A fact survives a rechunk. The question “what is the notice period” with the expected presence of “30 days” is still valid after you swap the embedding model, change the chunk size, and reindex from scratch. The chunk-level expectation is derived, not authored: you record which document the fact came from, and the retrieval metric asks whether any chunk from that document surfaced.

Size matters less than coverage of query types, and the honest number is that it grows from a few dozen questions to a few hundred, fed by two sources. Production queries that produced a thumbs-down or an escalated ticket get promoted into the set with the fact that should have been found. And every incremental update to the index runs the set before the swap, which is the other half of the point: the golden set is how you evaluate an index update, not only a code change. Reindexing a fifth of the corpus with a new document batch is a deploy, and it gets the same gate.

Which leaves the case where the correct answer is nothing. If the corpus does not contain the answer, the system should return no answer rather than the nearest paragraph dressed up as one, and product managers will fight you on this because an empty box looks like a broken product. The line I draw: an empty box with a reason beats a fabricated answer with a citation, every time, because the second one costs you the trust that makes the product usable. The signal is not raw cosine similarity, which is a poor proxy for “answerable.” It is a combination: whether the reranker’s top candidate clears a threshold you calibrated on the golden set’s unanswerable questions, and whether the generator, asked explicitly, can point to the span it is answering from. Tune that pair on the unanswerable slice of the set until the false-answer rate is one you can defend to the person whose name is on the product.

Guard every hop, not just the input

Most teams put a classifier on the user’s query and call the system guarded. That covers one boundary out of four. A retrieval pipeline has an ingest path, a retriever, a generator and a response, and each of them is a place where something malicious or merely wrong can enter. Off-domain queries are the easy case and belong at the input. The interesting attacks come through the corpus: a document that contains instructions to the model, which the retriever faithfully surfaces and the generator faithfully follows, because from where the model sits the chunk is just context. Data poisoning is the same attack run slowly. A document planted, or a public source edited, so that a biased answer becomes the best-matching evidence for a question somebody cares about.

The latency objection is legitimate. Three model calls per query on top of the generation is not a budget most products have. The answer is that most hops do not need a model. The tenant filter at retrieval is a WHERE clause. The ingest screen for injected instructions is pattern matching over a small vocabulary plus a review queue for what it flags. Stripping chunks of anything that looks like an instruction to the model is string processing. The one hop that benefits from a model is the output check, and it can run on a small, cheap model with a narrow question: does this answer contain a claim the context does not support. What you drop when latency bites is the input classifier, not the retrieval filter, because a bad input produces one bad answer and a missing tenant filter produces a breach.

Identity and time are retrieval keys

Two fields that the tutorial never has and every production corpus needs: who may see this document, and when it was true. Identity is the obvious one and it belongs in the retriever as a hard filter, not in the prompt as an instruction. If chunk metadata carries the tenant and the access group, and every query is filtered on the caller’s identity before similarity is even computed, then the model cannot leak a document it was never shown. Put the same rule in the prompt instead and you are trusting a language model to enforce access control, which it will do right until the query that phrases the request in a way you did not anticipate.

Time is subtler and more often missed. A large fraction of real questions are grounded in a period, not just in facts. What was the policy in March. What did the contract say before the amendment. Which version applied when this ticket was opened. A corpus that keeps only the latest document answers those wrong with total confidence, and a corpus that keeps every version without a valid-from field retrieves the wrong one at random. Give every chunk an effective date range, let the query carry a point in time, and filter on it the same way you filter on tenant. This is also the correct resolution to the near-duplicate problem from the first section: policy version three and version four are not duplicates to collapse, they are one document with two validity windows.

Both of these fields are why incremental index updates need the same care as a code deploy. Adding a batch of documents changes the neighbourhood of every existing chunk, and if the new batch carries wrong tenant tags or missing dates it corrupts the filters for everything near it. Run the golden set’s identity and time slices against the candidate index before it goes live, and keep the previous index until it passes.

Routing is an economics knob

At some point you will want the generation step to be cheaper, and the first reflex is to swap the model. A gateway such as OpenRouter makes that mechanically trivial: one endpoint in front of many models, with fallbacks handled for you and a cheaper option picked per request. That is fine as far as it goes, and for a lot of products it is the whole answer.

The more interesting version routes on the query rather than on price alone. A lookup question that the reranker answered with a single high-confidence chunk does not need the frontier model; it needs a small one that will quote the span accurately. A synthesis question over eight chunks from four documents with conflicting dates does. The user’s tier, the confidence of the retrieval stage, the number of candidate chunks and the detected query type are all inputs to a routing decision that you own. The only reason you can make it safely is the golden set: a model swap is a deploy, and the faithfulness slice tells you what the cheaper model broke before a customer does.

It is a recommender problem

Everything above is empirical configuration. Chunking strategy, chunk size, overlap, embedding model, similarity function, reranker, candidate depth, routing policy, thresholds for refusal. None of it has a closed-form answer and all of it interacts. That is not a research problem. It is an engineering problem with a familiar shape, and the shape is a recommender system. Recommender teams spent a decade discovering that the algorithm mattered less than the pipeline around it: candidate generation, ranking, filtering, and above all a feedback loop tied to an outcome the business cared about. The model was replaceable. The measurement was the product.

The objection is that recommenders had click data and RAG does not. Netflix could optimise session depth because every play was a label, and a question-answering product gets a thumbs-up on a sliver of its answers. True, and it is why the golden set carries so much weight here. But the implicit signals exist if you look. A follow-up question that rephrases the first one is a miss. A copied answer is a hit. A support ticket that does not get reopened after the bot answered it is the strongest label you will ever get. Wire those into the promotion path for the golden set and the loop closes: production tells you where retrieval failed, the set encodes it as a fact to find, and the next index update is gated on finding it.

Approach RAG that way and the tutorial’s questions stop being the questions. Not “which embedding model,” but “which change moved the outcome, and did the golden set catch it first.” That is the whole discipline, and it is the part the tutorial had no room for.