Why Do AI Chatbots Make Things Up?

Because a chatbot’s job is to produce text that fits, not text that’s verified. A large language model answers by continuing a pattern — given your question, what would a good answer look like? When it has genuinely absorbed the relevant facts, the fitting answer is also the true one. When it hasn’t, the fitting answer is a confident, well-formed guess. Nothing inside the model distinguishes between those two cases, and nothing makes it sound less sure in the second one. That’s what people mean by hallucination: fabricated content delivered in the same tone as fact.

Understanding why this happens is the fastest route to knowing when to trust a chatbot — and it explains why retrieval exists at all.

The machine is a pattern completer, not a database

Picture the difference between two ways of answering “when was this building constructed?”

A database looks up a row. Either the row exists or it doesn’t. If it doesn’t, you get nothing.

A language model does something else entirely. It has read millions of sentences about buildings and construction dates, and it produces the words most likely to follow your question given everything it absorbed. If the specific building was well documented in its training text, the likely continuation is the right year. If it wasn’t, the likely continuation is still a year — because that’s the shape answers to this question take.

There is no empty result. The model always has something to say. This is the root of the whole problem: a system that always produces output will produce output even when it has nothing.

Fluency and accuracy are separate skills

Human speech gives us useful signals. People hedge, hesitate, trail off, say “I think it was the eighties?” We’ve spent our lives reading those cues.

Language models learned to write well from text that was, on the whole, written by people who knew what they were talking about. So the model absorbed the style of confident expertise as a general property of good writing. It applies that style uniformly — to facts it has nailed down and to material it’s improvising.

The result is that the usual human reliability cues are absent. A fabricated citation has the same crisp formatting as a real one. An invented statistic arrives with the same decimal precision as a measured one. You cannot tell from the prose, which is genuinely unsettling the first time you catch it.

The five things chatbots make up most often

Patterns worth memorizing, because they predict where to be suspicious:

Citations and references. Paper titles, authors, page numbers, court cases, URLs. The model has seen thousands of correctly formatted references and can generate the format perfectly while the content is invented. This is the single most notorious failure mode.

Numbers. Percentages, dates, prices, measurements. A plausible-looking number is easy to produce and impossible to verify by inspection.

Obscure specifics. Small companies, minor historical figures, niche products, local details. The less the training text covered something, the more the model fills gaps.

Anything after the cutoff. Models are trained up to a point in time. Ask about later events and, unless the product searches for you, you get either a refusal or confident invention.

Details about you. Your company’s policy, your account, your codebase. The model never saw any of it, but the question implies an answer exists, so it constructs one.

Why “just say you don’t know” is harder than it sounds

The obvious fix — have the model decline when unsure — runs into two real obstacles.

First, the model’s sense of its own uncertainty is imperfect. It can produce a confident-shaped continuation without any internal flag saying “this part is invention.” Some uncertainty signal exists inside these systems, but it’s noisy and doesn’t map cleanly onto “this specific claim is fabricated.”

Second, declining is penalized in practice. A model tuned to be maximally helpful learns that answers get better reactions than refusals. Push too hard the other way and you get an assistant that shrugs at reasonable questions, which users hate just as much. Builders are constantly negotiating that trade-off, and different products land in different places.

How retrieval attacks the problem

This is exactly the gap RAG targets. If the relevant passage is sitting in front of the model, it doesn’t have to reconstruct the fact from statistical impressions — it can read it. Open book instead of closed book.

That helps a great deal, and it adds something even more valuable: checkability. A grounded answer can point at its source, and you can look. An answer from memory offers nothing to inspect.

But retrieval is not a cure, and anyone selling it as one is overselling. If the search step returns the wrong passage, the model answers wrongly from a real document — which is arguably harder to catch, because the citation is genuine even though it doesn’t support the claim. And a model handed context can still drift back into its own memory mid-sentence. We go through those failure modes in When does RAG give wrong answers?

What you can actually do about it

Practical habits, in rough order of usefulness:

Prefer answers with sources, then open the sources. Not the summary of the source — the source. Verifying is a click.

Ask “what are you basing that on?” A grounded system names a document. A guessing one produces vague attribution or quietly changes its story.

Be more suspicious as specificity rises. A general explanation of how mortgages work is likely fine. A precise interest rate for a named lender on a named date is exactly the kind of claim that gets invented.

Watch for the confidence-obscurity mismatch. Fluent, detailed answers about something genuinely obscure are the classic warning sign.

Never accept a reference without checking it exists. This one habit prevents most public embarrassments involving chatbots.

The takeaway

Chatbots make things up because they generate plausible text rather than retrieve verified facts, and because fluent writing style is applied evenly whether the underlying knowledge is solid or absent. Retrieval-augmented generation reduces the problem by putting real sources in front of the model and letting it show its work — a large improvement, not an immunity. Treat confident prose as a writing style, not evidence.