A production RAG system must retrieve relevant, authorized evidence and show where an answer came from. Start with source quality, permission boundaries and an evaluation set. Model choice matters, but weak retrieval and missing observability are more common causes of failure.
RAG is an evidence system
Retrieval-augmented generation, or RAG, gives a language model selected information at answer time. That sounds simple, but the production problem is not attaching a vector database to a chatbot. It is building an evidence pipeline that retrieves the right material, respects access rules and lets a reader judge where an answer came from.
A convincing demo can succeed with a handful of clean documents. A real system must handle duplicates, stale policies, tables, scans, conflicting versions and questions whose answer is not present. Treat the model as the final stage of a search and evidence product, not as a substitute for one.
Design retrieval before generation
Start with representative questions and identify the passages a knowledgeable employee would need to answer them. This becomes a small evaluation set. Test keyword, semantic and hybrid retrieval against it before tuning prompts. For specialised vocabularies, hybrid search and metadata filters often outperform embeddings alone.
Chunk boundaries should follow meaning rather than an arbitrary character count. Preserve headings, document identity, dates and relationships between tables and explanatory text. Reranking can improve the final context set, but it cannot repair missing or badly parsed source material.
Permissions and citations are architecture
Apply authorization during retrieval. Filtering documents after they have entered the prompt is too late because restricted information has already crossed a boundary. Carry the user's identity and entitlements into the search layer, and test access changes as carefully as answer quality.
Citations should resolve to the exact source passage, not merely a document homepage. Show document title, revision date and enough surrounding context for verification. When evidence is weak or contradictory, the system should say so. A useful refusal is more trustworthy than a fluent invention.
Evaluate the complete answer path
Measure retrieval recall, citation correctness, groundedness and task usefulness separately. A model can produce a reasonable answer from the wrong passage; an excellent passage can still be summarised badly. Keep failed questions as regression cases and review evaluation changes whenever sources, chunking, embedding models or prompts change.
In production, log document versions, retrieval scores, latency and the evidence supplied to the model, subject to privacy rules. Monitor unanswered queries and user corrections. Those signals reveal content gaps and retrieval problems that an offline benchmark will miss.
Frequently asked questions
What should a team build first in a RAG project?
Build a representative question set and a traceable ingestion-and-retrieval path. Prove that the system can find authorized evidence before investing heavily in prompts or model selection.
Does a vector database guarantee accurate answers?
No. Accuracy depends on source quality, parsing, chunking, retrieval, permissions, context assembly and generation. A vector database addresses only part of that chain.
Further reading
Explore official documentation for the tools and architecture patterns discussed in this guide.