Production RAG

The fuller explanation

By this point retrieval quality is a solved problem in the demo: documents are chunked sensibly, search is hybrid, results are reranked, and there are numbers saying whether a change helped. Production is where the remaining work lives, and almost none of it is about relevance.

Ingestion stops being a script and becomes a pipeline with a schedule, because the corpus changes. Pages are edited, documents are archived, someone deletes a file. Each of those has to propagate to the index, and deletions are the ones that get skipped, which is how a system confidently quotes a policy that was withdrawn months ago. Permissions arrive next, and they cannot be bolted on afterwards: the retriever must filter by what this user is allowed to see, at query time, because a chunk that reaches the context window has already leaked. Answer caches inherit the same problem, since a cached answer built from documents one user could read must not be served to another.

Then cost and latency, which are mostly decided by how much context you send and how many models you chain to produce one answer. The useful instinct is that a production RAG system is a data pipeline with a language model at the end, and it fails in the ways data pipelines fail.