Ask a general AI chatbot about your company’s refund policy and you will get a confident, polite, completely invented answer. The model has never seen your policy. It only knows what was in its training data, and that data stops at a fixed date.
Retrieval-augmented generation, or RAG, is the standard fix. Before the model writes anything, the system searches a collection of documents you control and hands the most relevant passages to the model along with the question.
Where the idea came from
The name comes from a 2020 research paper by Patrick Lewis and colleagues, accepted at the NeurIPS conference. Their system paired a language model with a searchable index of Wikipedia. The model’s built-in knowledge was one kind of memory; the index was a second kind that could be read and updated separately. That split is still the core of every RAG system today.
How it works, step by step
Picture an HR assistant answering the question: “How many vacation days do I get after two years?”
- Ahead of time, the employee handbook is cut into short chunks, often a paragraph or a section each.
- Each chunk is turned into an embedding, a long list of numbers that captures what the text is about. Chunks about leave policy end up numerically close to each other.
- When the question arrives, it is turned into an embedding too, and the system finds the chunks closest to it.
- Those few chunks, perhaps the section on paid leave, are pasted into the prompt with an instruction like “answer using only this material”.
- The model writes its answer, and the app can show which handbook section it came from.
Why not just retrain the model?
You can fine-tune a model on your documents, but it is slow and expensive, and you have to do it again every time the documents change. With RAG, updating the knowledge means re-indexing one file. It also makes answers checkable: if the assistant cites section 4.2 of the handbook, a person can open section 4.2 and confirm it.
Where RAG goes wrong
Almost every RAG failure is a search failure. If the retrieval step hands the model the wrong paragraph, the model will answer from the wrong paragraph, fluently. The usual culprits:
- Chunks cut in awkward places, so a rule and its exception end up in different pieces.
- Questions phrased differently from the documents. Someone asks about “time off”; the handbook only says “annual leave”.
- Old and new versions of a document both sitting in the index.
- Too many chunks stuffed into the prompt, burying the one that matters.
Teams that get good results tend to split documents along their headings, combine keyword search with embedding search, store dates and versions with each chunk, and keep a list of 30 to 50 real questions with known correct answers to test every change against.
Doesn’t a huge context window make RAG pointless?
Newer models can read hundreds of thousands of words in a single prompt, so a fair question is why not paste the whole handbook in every time. For a small, stable document set, that can work fine. It stops working as the collection grows. Sending every page with every question costs more, takes longer to answer, and models still tend to miss details buried in the middle of a very long prompt. Retrieval keeps the prompt short and focused, and it scales to thousands of documents that would never fit in one request.
In practice the two ideas are combining: systems retrieve a generous set of relevant sections and let the larger context window absorb them, instead of squeezing everything into three tiny chunks.
Starting small
You don’t need a dedicated platform to try this. Most cloud providers and AI vendors now offer a managed “knowledge base” feature, and open-source libraries handle chunking and search in a few lines of code. A sensible first project:
- Pick one document set people ask about constantly, such as product FAQs or internal IT guides.
- Collect real questions from support tickets or chat logs, along with the answers an expert would give.
- Build the simplest pipeline that works, and score it against those questions.
- Only add complexity, like re-ranking or query rewriting, when a specific failing question shows you need it.
Always show the source passage next to the answer. It builds trust with users and turns every wrong answer into an easy bug report.
What we think
RAG is the most practical way to make AI useful on your own documents, and for most companies it beats fine-tuning on cost alone.
But treat it as a search project first and an AI project second. When an answer is wrong, look at what was retrieved before you blame the model. Nine times out of ten, that is where the problem is.