How Does an AI Know Which Documents Are Relevant?

It compares meanings rather than words. Before you ever ask anything, every passage in the document collection is converted into a long list of numbers that represents what it’s about. Your question gets the same treatment. Then the system finds the passages whose numbers are closest to your question’s numbers, and those become the “relevant” ones. Nothing understands your question in the way a colleague would — it’s a distance calculation, and that’s both why it works surprisingly well and why it sometimes picks something baffling.

Let’s build the intuition properly, because this one mechanism explains most of RAG’s behavior.

The map of meaning

The mental model to hold onto is a map. On an ordinary map, position encodes geography: things near each other are physically close. On this map, position encodes meaning: things near each other mean similar things.

“Dog” sits near “puppy” and “canine.” “Bank” is awkward, because it means two things, and lands somewhere between finance and rivers — which is exactly the kind of ambiguity that causes real retrieval mistakes. A sentence about cancelling a subscription sits near other sentences about cancelling subscriptions, even if one says “terminate my plan” and shares no words with the other.

The numbers that place text on this map are called embeddings. An embedding is just the coordinates of a piece of text on the map of meaning, produced by a model trained specifically to put similar meanings in similar places. Real ones have hundreds or thousands of coordinates rather than two, which is impossible to picture but changes nothing conceptually — near still means similar.

What happens when you ask a question

Four things, in order:

Your question becomes coordinates. The same embedding model that processed the documents processes your question. Using the same model for both is essential; two different maps aren’t comparable.

The system looks for nearby passages. It searches the collection for the coordinates closest to your question’s. Special databases exist to do this quickly over millions of items — that’s what a vector database is for, as covered in What does a vector database do in RAG?

A shortlist comes back, ranked. Closest first. Usually a handful.

Sometimes a second pass re-sorts them. Many systems then run a slower, more careful comparison over just those few candidates, since it’s affordable on ten items and impossible on ten million.

Then the shortlist goes into the prompt and the model writes the answer. “Relevant” means nothing more mystical than “closest on the map, according to the shortlist.”

Why meaning-matching beats keyword matching

Traditional search matches words. Type “cancel subscription” and it finds documents containing those words. If your help article says “how to terminate a recurring plan,” keyword search misses it and you get nothing.

Meaning-based search handles that. Paraphrase, synonyms, and roundabout phrasing all land in roughly the same neighborhood, so a question about “getting my money back” can find a passage about “refund eligibility.”

This is why RAG chatbots feel more forgiving than a search box. You can ask badly and still get the right page.

Why keyword matching is still in the mix

Meaning-based search has a specific weakness: precision on exact strings. Product code “XR-4471B” and “XR-4471C” mean nearly the same thing to an embedding model — both are product codes, similar context, nearly identical characters — so they sit almost on top of each other on the map. For you, they’re different products.

The same problem hits names, dates, version numbers, error codes, and legal citations. Anything where being exactly right matters is exactly where “close in meaning” fails.

That’s why most serious systems run both kinds of search and combine the results: meaning-based search for phrasing flexibility, keyword search for exact terms. You get the forgiveness of one and the precision of the other.

Why it sometimes picks the wrong thing

Now the honest part. Several predictable failure patterns follow directly from the mechanism.

Similar-sounding but distinct topics collide. The 2023 and 2026 versions of a policy are nearly identical in meaning. The map can’t tell you which is current — that’s what document dates and filters are for, if someone set them up.

Vague questions land in vague places. “Is this covered?” contains almost no meaning to match on. Coordinates for a near-contentless question point somewhere unhelpful, and you get a random-feeling shortlist.

Splitting decides what’s findable. Documents are cut into pieces before embedding. If the answer spans a cut, no single piece contains it, and no search technique can retrieve something that isn’t intact anywhere.

Popular topics crowd out rare ones. If a hundred passages discuss expenses and one discusses expense appeals, a question about appeals may fill its shortlist with general expense material.

Nothing relevant exists, but you still get results. Similarity search always returns the closest items — closest is not the same as good. Without a quality threshold, an unanswerable question produces a shortlist of irrelevant passages, and then the model tries to answer from them.

That last one causes more confusion than any other. Retrieval doesn’t return “no results”; it returns “the least bad options I have.”

What this means for asking better questions

Because relevance is meaning-matching, you can help it:

• Use the vocabulary your documents use. If internal docs say “PTO,” asking about “PTO” beats asking about “holidays.”
• Include distinguishing detail. “Refund policy for annual plans” gives far more to match on than “refunds.”
• Ask one thing at a time. A three-part question retrieves a muddle; three questions retrieve three good shortlists.
• Name the timeframe or document if you know it. “In the 2026 handbook…” narrows the neighborhood.
• Rephrase if the answer looks off. Different words, different coordinates, different shortlist.

The takeaway

An AI decides which documents are relevant by turning text into coordinates on a map of meaning and grabbing the nearest neighbors to your question — usually blended with old-fashioned keyword matching to catch exact terms. It’s not comprehension; it’s geometry. That explains both the pleasant surprise of being understood despite clumsy phrasing and the occasional maddening result, because “nearest” and “correct” are not the same thing.