RAG explained: how a chatbot answers from your documents

What to know

• RAG stands for retrieval-augmented generation: find the relevant passages first, then have the model answer using them.
• It lets a model use information it was never trained on, such as your company's files or last night's news.
• The answer can only be as good as the passages that were retrieved.

A language model knows what was in its training data and nothing after. It has never seen your company handbook or yesterday's sales report. Retrieval-augmented generation, or RAG, is the standard way around that.

The idea in one sentence

Before the model answers, a search step finds the passages most likely to contain the answer and places them in front of the model along with your question.

It is the difference between a closed-book exam and an open-book one.

The steps

Ahead of time, once:

1. Documents are split into chunks, usually a few paragraphs each.
2. Each chunk is converted into a list of numbers, called an embedding, that represents its meaning. Passages about similar things end up with similar numbers.
3. The chunks and their embeddings are stored in a searchable index.

Each time you ask a question:

1. Your question is converted into an embedding the same way.
2. The system finds the chunks whose embeddings are closest to it.
3. Those chunks are added to the prompt, with an instruction to answer from them.
4. The model writes the answer, ideally pointing to the passages it used.

Why it is popular

• Fresh information. Update the documents and the answers update. No retraining.
• Checkable answers. The system can show which passages it relied on.
• Access control. Search can be limited to documents a given person is allowed to see.
• Cost. It is far cheaper than training a model on your data.

Where it fails

Most RAG failures are search failures.

• The right passage was not retrieved. The model then answers from general knowledge or guesses. This is the most common problem.
• Chunks cut in the wrong place. A table split in half, or a rule separated from its exception, produces confident wrong answers.
• Outdated or conflicting documents. If the index holds three versions of a policy, the model may quote the wrong one.
• Questions that need the whole document. "Summarize every complaint from last quarter" cannot be answered from five retrieved chunks.

RAG lowers the rate of made-up answers. It does not remove them.

How to judge a RAG system

• Does it cite its sources, and do the citations say what it claims?
• Ask something the documents do not cover. A good system says it does not know.
• Ask something where two documents disagree and see whether it notices.

RAG and fine-tuning

They solve different problems. RAG gives a model knowledge to look up. Fine-tuning changes how a model behaves, such as its tone or its output format. Many systems use both.

The retrieved passages have to fit in the model's working memory. For how that limit works, see what a context window is.

Spotted an error? Tell us and we will fix it.