RAG implementation: what the tutorials skip and production teaches

Retrieval-augmented generation is the most-deployed pattern in applied AI for a simple reason: it lets a language model answer from your documents instead of its training data. The basic tutorial version takes an afternoon — embed documents, store vectors, retrieve top matches, stuff them into a prompt. The production version is where projects live or die, and it's governed by problems the tutorials never mention.

When a RAG system gives a bad answer, teams instinctively blame the model and start prompt-tuning. In our experience, the majority of RAG failures are retrieval failures: the right passage was never fetched, so no prompt could save the answer. Before touching prompts, measure retrieval directly — for a set of real questions, is the passage containing the answer in the retrieved set at all? If your retrieval hit rate is 60%, your answer quality ceiling is 60%, and everything downstream is decoration.

How you split documents determines what can be retrieved. Fixed-size chunks slice tables in half and orphan headings from their content. The chunking strategy has to match the documents: contracts want clause-level chunks with section headers attached, support articles want question-sized chunks, spreadsheets often shouldn't be chunked as text at all. Attach metadata (source, section, date) to every chunk — you will need it for both filtering and citations.

Semantic search misses exact identifiers — product codes, error numbers, names — that keyword search catches trivially, and keyword search misses paraphrases that embeddings handle easily. Production systems that hold up almost always run hybrid retrieval (vector + keyword) with a reranking step on top. The reranker is the highest-leverage component most first builds skip: retrieving 30 candidates and reranking to the best 5 dramatically outperforms retrieving 5 directly.

A RAG system that answers without citing its sources is unauditable, and in any serious business context that's disqualifying. Build citation into the design from the start: every claim in the answer should be traceable to a retrieved chunk, and the UI should let users see the source. This isn't just trust theater — citations are also your best debugging tool, because a wrong answer with citations shows you immediately whether retrieval or generation failed.

You cannot improve what you don't measure, and RAG has two things to measure: retrieval quality (did the right chunks come back) and answer quality (was the response correct, complete, and grounded in the sources rather than hallucinated around them). Build a golden dataset of real questions with known-correct answers and source passages — even 50 examples transforms development from guesswork into engineering. We covered the general discipline in our guide to testing AI agents, and RAG is where it pays off fastest.

Document sync: your knowledge base changes; if the index doesn't update, the system confidently answers from stale data — decide the refresh strategy on day one. Permissions: if different users can see different documents, retrieval must respect that per query, and bolting access control on later is painful. Cost: context length is money; retrieving less, better-ranked content beats stuffing the window. And failure behavior: the system needs a defined answer for "the documents don't cover this" — the worst RAG systems are the ones that never say "I don't know."

RAG rewards engineering discipline far more than model sophistication. Retrieval measurement, deliberate chunking, hybrid search with reranking, citations, evaluation datasets, and boring operational hygiene — that's the difference between the afternoon demo and the system people trust. If you're scoping a RAG build over your own document base, our AI implementation team builds these production concerns in from the first sprint rather than retrofitting them after launch.