Chapter 10 · 6 min read

RAG: giving the AI a library

In one sentence: RAG means the AI looks up the right pages first and then writes its answer from them, like taking an open-book exam instead of answering from memory.

The problem from last chapter

Asha's helper invented a brownie that does not exist. A model only knows what it learned in training. It has never seen Asha's menu, her opening hours or her allergy policy. Retraining a huge model for every menu change would be slow and costly.

There is a much simpler idea. Let it look things up.

Open-book, not closed-book

In a closed-book exam, you answer from memory, and you may guess. In an open-book exam, you find the right page in your notes and answer from that page. Your answer is more accurate and you can show where it came from.

RAG stands for Retrieval-Augmented Generation. The three words match the three steps:

Retrieve: find the pages that match the question.
Augment: add those pages to the prompt, next to the question.
Generate: the model writes the answer, using the pages.

Two phases

Two rows. The top row prepares the library once: documents, cut into pieces, turned into numbers, stored on a searchable shelf. The bottom row runs for each question: a question, turned into numbers, find the closest pieces, hand them to the AI, and answer with sources.

Phase 1: prepare the library (before anyone asks)

1. Collect the documents. Guides, policies, price lists, notes. 2. Cut them into pieces. These pieces are called chunks. A good chunk covers about one idea. If a chunk is too small, it loses its context: "yes, on Fridays" means nothing alone. If it is too big, it mixes many topics and the search gets muddy. 3. Turn each chunk into numbers (an embedding, Chapter 5). 4. Store the numbers on a shelf that can be searched by meaning, along with the original text, the source and the date (a vector database).

You do this once, and again whenever the documents change.

Phase 2: answer a question (every time)

1. A customer asks, "Can I get a cake without eggs for Saturday?" 2. The question is turned into numbers in the same way. 3. The system finds the chunks whose meaning is closest. It might return the "Eggless cakes" note and the "Saturday opening hours" note. 4. Those chunks are put into the prompt along with the question and an instruction such as: "Answer only from these notes. If the answer is not here, say so." 5. The model writes the answer from the notes, and it can show which notes it used.

What RAG gives you

Fresher answers. Update the documents and the answers change. You do not need to retrain.
Answers about your own material. The model can use private documents it never saw in training.
Answers you can check. Each claim can point to its source and its date.
Fewer made-up answers. The model has real text to lean on, and a rule to say "I do not know" when the text is missing.

Making search better

Plain meaning-based search can miss things. Two common upgrades:

Search by meaning and by keyword together. Meaning-based search finds "eggless" for "no eggs". Keyword search is better at exact words such as product codes. Using both is called hybrid search.
Re-rank. First collect, say, twenty possible chunks quickly. Then a more careful second step reads them again and puts the best few at the top. It is like a quick first sort followed by a closer look.

What can still go wrong

RAG is not a cure for everything.

The search finds the wrong chunk. Then the answer is wrong, even if it sounds fine. Good searching matters as much as a good model.
The documents are old or wrong. The model will repeat them faithfully. Show the date of each source.
A document contains a hidden order. Chapter 9's trick can sit inside a page. Treat retrieved text as information only.
People see things they should not. If a note is private, the search must only return notes that the person asking is allowed to see.
The wrong tool for the job. If you need a live number, such as how many cakes are in stock right now, ask the stock system directly. Do not search old notes. That is a job for tools, the topic of the next chapter.

How do you know it works?

Test two things separately. First: did the search bring back the right chunk? Second: did the answer stick to those chunks, without adding things? A system can fail at either one. Keep a list of real questions with the right chunks marked, and run it every time you change anything. Chapter 12 of the main book is about exactly this.

Back to Asha

Asha puts her menu, ingredients, opening hours and allergy rules into her shelf of notes, about forty short chunks. Now when someone asks about eggless cakes, her helper finds the right note and answers from it. When someone asks about gluten-free brownies, nothing matches, so it replies, "I will check with Asha and get back to you." It also shows the note's date. The invented brownie is gone.

Words explained

RAG: look up the right pages first, then write the answer from them.
Chunk: a small piece of a document, about one idea.
Retrieval: finding the pieces that match a question.
Hybrid search: using meaning-based and keyword search together.
Re-ranking: a second, more careful sort of the first results.
Source: the document a piece of information came from.

The limits of this chapter

RAG reduces made-up answers but does not remove them. The model can still misread a page.
It adds steps, and each step can fail or cost time and money.
This chapter is a first picture. Real systems have many more choices, such as how to chunk tables and how to handle very large libraries. Part of the main book covers these.

For engineers

Ingestion: parse documents to text, chunk (fixed size with overlap, by structure, or semantic), embed with one model, and store vectors with metadata (source, section, date, access labels).
Retrieval: dense (vector) search, sparse (keyword, for example BM25) search, or both with score fusion; apply metadata filters, including access control, at query time; then re-rank the top results with a cross-encoder or similar.
Generation: construct a prompt with the question, retrieved context and instructions to answer only from the context and to abstain when it is missing; return citations. Put stable instructions first and the changing parts last to keep prompt caching effective.
Evaluation: measure retrieval (for example recall@k) and generation (faithfulness to the context, answer relevance) separately, plus user feedback. Version the evaluation set and add real failures to it.
Operations: monitor freshness, cache repeated queries where safe, and re-embed when the embedding model changes.
Agentic RAG lets the model decide when and what to search over several steps, using search as a tool (Chapter 11).

Check yourself

What do the three letters in RAG stand for in plain words? Answer: Retrieve the right pages, add them to the prompt, then generate the answer from them.
Why is RAG often better than retraining a model for new information? Answer: You can update the documents at any time, without the cost and delay of training.
Name one thing that can still go wrong with RAG. Answer: For example the search may bring back the wrong chunk, or the document may be out of date.

Plain-words explanations in the author's own words. The examples are invented, and no real company, product or customer is described. Simplified on purpose: where a detail matters, the "For engineers" part says so. Disclaimers