How Does RAG Work?
RAG works in four steps that happen in about a second, every time you ask a question: your question becomes a search, the search returns a few relevant passages, those passages are pasted into a hidden instruction alongside your question, and the AI writes an answer from them. Everything else — embeddings, vector databases, chunking — is machinery serving those four steps. If you understand the sequence, you understand RAG.
Let’s walk through it the way it actually happens, from the moment you hit Enter.
Before you ask anything: the library gets built
One thing has to happen in advance. Somebody takes the documents the system should know about — a handbook, a product catalog, ten years of support emails — and prepares them for searching.
Preparing means two things. First, long documents get split into smaller pieces, because “the relevant part of this 300-page manual” is more useful than the whole manual. Second, each piece gets indexed so it can be found quickly later. This one-time (well, periodically repeated) preparation is why a RAG chatbot can answer instantly instead of reading everything from scratch.
Think of it as a librarian cataloguing the shelves before opening hours. Nothing is answered yet; the books are just made findable.
Step 1: Your question becomes a search
You type “how many vacation days do new hires get?” The system’s first job is turning that into something searchable.
The most common approach uses embeddings — a way of converting a piece of text into a long list of numbers that captures its meaning rather than its exact words. Two texts about the same idea end up with similar numbers even if they share no vocabulary. The handy mental picture is a map where distance means difference in meaning: “vacation days” and “paid time off” sit close together, “vacation days” and “fire extinguisher” sit far apart.
Your question gets converted into that numeric form, and so was every passage in the library during cataloguing. Now finding relevant material is just finding nearby points on the map.
Many systems also run an old-fashioned keyword search at the same time, because exact terms — a part number, a person’s name, an error code — matter and meaning-based search can be fuzzy about them.
Step 2: A few passages come back
The search returns a ranked shortlist: typically a handful of passages, not hundreds. Perhaps three, perhaps twenty, depending on how the system was built.
Why so few? Because the model can only consider a limited amount of text at once, and because irrelevant passages actively hurt — they crowd out the good material and give the model a chance to answer from the wrong source. Retrieval is a filter, and a filter that lets everything through isn’t doing its job.
Notice what has not happened yet: no AI has written anything. So far this is a search engine with an unusual query format.
Step 3: The hidden prompt gets assembled
This is the step nobody sees and the one that makes RAG RAG. The system builds a single block of text containing:
- an instruction, roughly “answer the user’s question using only the material below, and say so if the material doesn’t cover it”
- the retrieved passages, often with their source titles attached
- your original question
That whole bundle goes to the language model as one message. From the model’s perspective, you handed it the source documents yourself. It has no idea a search happened.
This is why the open-book exam analogy fits so well. The model is not recalling facts from study; it’s reading the passages in front of it and writing an answer. Same student, different exam conditions.
Step 4: The model writes the answer
Now the large language model — the text-generating AI at the center of tools like ChatGPT — does what it always does: produces a fluent answer one piece at a time. The difference is that the specific facts it needs are sitting right there in the prompt, so it doesn’t have to rely on whatever it absorbed during training.
Well-built systems also ask the model to cite which passage each claim came from, which is where those little source links in chatbot answers come from. Citations matter for a practical reason: an answer you can trace is an answer you can check.
What makes the difference between good and bad RAG
The four steps are simple. Doing them well is not, and almost all the difficulty concentrates in two places.
Retrieval quality is the ceiling. If the passage containing the answer never makes the shortlist, no amount of clever writing saves the answer. The model will either say it doesn’t know or — worse — improvise from whatever it did get. Most disappointing RAG systems are search problems wearing an AI costume.
How documents were split matters more than beginners expect. Cut a document in the wrong place and the answer ends up straddling two pieces, with neither piece making sense alone. A table separated from its column headings becomes numbers with no meaning.
Both of those are invisible from the outside, which is why two chatbots built on the same model and the same documents can feel completely different to use.
What doesn’t happen
Two clarifications that save a lot of confusion:
The model does not learn from your documents. Nothing about the model changes. The documents are shown to it, used once, and forgotten when the conversation ends. That’s the opposite of fine-tuning, which does adjust the model itself — see RAG vs. fine-tuning: which do you need?
Retrieval happens per question, not per conversation. Ask a follow-up and a fresh search usually runs. This is why a good system can handle “what about contractors?” — it re-retrieves with the new context in mind.
The takeaway
RAG works by searching a prepared library for passages relevant to your question, quietly pasting them into the prompt, and asking a language model to answer from them. Retrieve, augment, generate — in that order, every time. The magic isn’t the AI part; it’s the fact that the AI is answering with the right page already open in front of it. For the plainest possible version of the concept, see What is RAG in simple terms?