A retrieval-augmented generation demo takes an afternoon. A production system takes months, and the gap is not model quality. It is a specific set of problems that a demo corpus, a demo question set and a demo user never expose.
Almost all of them reduce to two things: whether retrieval returns the right source, and whether anyone can tell. The first is an engineering problem with known fixes. The second is a measurement problem, and the absence of measurement is what turns the first into a quarter of guesswork.
Here is what actually breaks, in roughly the order teams encounter it.
The corpus is not a corpus
In the demo, documents were chosen. RAG development in production starts at the other end: the system is pointed at a document store that has accumulated for a decade.
Inside it: superseded versions sitting beside current ones, drafts that were never marked as drafts, documents contradicting each other because policy changed, and near-duplicates in a dozen places.
Retrieval does its job and returns the superseded document, because it is textually a better match than the current one. The answer is confidently wrong and traceable to a real source, which makes it harder to dismiss than a hallucination.
What to do. Before any retrieval work, decide what is in scope and what is not, and encode that as metadata the retriever filters on. A smaller authoritative corpus beats a large comprehensive one every time. If a document is superseded, either exclude it or mark it so the answer can say so. Anything the current stack cannot read is a separate job that comes first: a folder of legacy XML exports has to be transformed into something queryable before indexing it means anything.
Chunking destroys the thing you needed
Documents get split to fit retrieval. The split puts the condition in one chunk and the exception in another, and retrieval returns one of them.
This is the most common cause of answers that are right in substance and wrong in effect: the rule is correctly stated and the exception that applied is missing.
What to do. Chunk on document structure rather than character count where structure exists, keep headings with their content, and overlap deliberately. Test specifically for conditional content: find the rules in your corpus that have exceptions, and check that queries about them retrieve both parts.
Questions do not look like documents
Users ask short, elliptical, jargon-heavy questions. Documents are written in formal prose. The vocabulary barely overlaps.
Demo questions were written by the person who read the documents, so they used the documents’ words. Real questions use the user’s words, including internal abbreviations that appear nowhere in the corpus.
What to do. Collect real questions before building, not after. Where internal terminology is used, build a mapping and expand queries with it. This is unexciting lexical work and it moves retrieval quality more than changing embedding models.
Access control was not in the demo
In the demo everyone sees everything. In production the corpus contains documents that specific users must not see, and the retriever has no concept of who is asking.
This is the problem most likely to stop a rollout outright, and it is the hardest to retrofit, because the permission model has to be present at retrieval time rather than applied to the answer.
What to do. Decide this before indexing. Carry each document’s access metadata into the index and filter at query time, as part of the retrieval request rather than after it. Filtering afterwards leaks through the generated answer even when the citation is withheld. The model worth copying is attribute based access control, where authorization is decided by evaluating attributes of the subject, the document and the requested operation against policy, not by a list bolted onto the answer.
There is no way to tell whether it is getting better
A team changes the chunking, the embeddings or the prompt, and someone says it seems better. There is no measurement, so every change is a matter of opinion and regressions ship invisibly. It is the failure that A/B testing infrastructure exists to prevent: with no structure to measure against, a lift is a story rather than a result.
What to do. Build an evaluation set before the first optimisation: real questions with a known correct source document and a known acceptable answer. A hundred items is enough to detect meaningful change. Separate two measurements, because they have different fixes: did retrieval return the right source, and did generation use it correctly.
What to measure, and what a bad number points to
| Measure | What it answers | Where the fix sits |
| Retrieval hit rate at k | Did the right source appear in the top results at all | Chunking, metadata and vocabulary, not the model |
| Citation support rate | Does the cited source actually contain the answer given | Generation, not retrieval |
| Superseded-source rate | How often answers cite a document that is no longer current | Corpus scope and the re-indexing schedule |
| Permission misses | Was any answer built on a document the asker may not see | Filtering applied after retrieval instead of inside it |
| Rephrase rate | How often one user asks the same thing twice in a session | A retrieval miss, which arrives later as a bill |
| Cost per resolved question | What one answered question costs, rather than one query | Everything above, in a single number |
The last row is the one to put in front of whoever owns the budget. Cost per query can be pushed down by shortening answers and retrieving less context, which makes the chart look better while the system gets worse. Cost per resolved question only falls when retrieval improves, because a question answered on the first attempt stops producing follow-ups.
Freshness becomes a support problem
The corpus changes. A policy is updated, a price list is replaced, a procedure is revised. The index does not know.
Users notice quickly, and the first wrong answer about current policy costs more trust than fifty correct answers earn.
What to do. Make re-indexing a scheduled, monitored job rather than a manual one, and show the user the source and its date with every answer. A visible date lets the user catch staleness themselves, which is far cheaper than guaranteeing freshness.
Citations are harder than they look
Showing the source is the single feature that most increases user trust, and users check them. When a citation points to a chunk rather than a readable location in the original document, trust falls rather than rises.
What to do. Preserve enough positional metadata through the pipeline to link back to a page, section or heading in the original. Decide this early; it is painful to add once the index exists.
Cost arrives later than expected
Demo volumes are trivial. Production volumes are not, and the cost per query is a function of how much context is retrieved, how long the answers are, and how often users rephrase because the first answer missed.
That last factor is the one that surprises teams: poor retrieval shows up as a cost problem, because every bad answer produces two more queries.
What to do. Instrument cost per resolved question, not per query. It is the number that correlates with both spend and quality.
The honest sequence
Teams that get a RAG system into real use tend to follow a similar order: define the authoritative corpus, settle the access model, collect real questions, build the evaluation set, then iterate on chunking and retrieval against that set, with citations and dates visible from the first release.
The same order holds when the system is bought rather than built. Enterprise AI search tools arrive with connectors to the places documents already sit, and they answer instead of returning a list of links, but they inherit whichever corpus and permission model they are pointed at. Buying shortens the engineering, not the scoping.
Teams that start by comparing embedding models usually spend a quarter on measurements that did not matter, because their corpus contained three versions of the answer.
Three answers shape a retrieval project more than any model choice. Which documents are authoritative. Who may see them. How next month’s version will be compared with this one. Settle those three and the rest is ordinary engineering; leave them open and no model work will make the answers trustworthy.
Comments 0
No comments yet. Start the conversation!
