What is an AI hallucination?
The invented case citation reads exactly like the real ones, which is why it reaches the customer before anyone checks it.
Hallucination
Confabulation
A hallucination is output from an AI model that reads as fluent and confident but has no real source behind it. Invented citations, plausible wrong numbers, and policies the model pieced together rather than read all count.
Language models predict likely next words. They are not looking anything up unless you build the lookup. A fabricated answer and a correct one are produced by the same process, so the model has no internal signal that one is wrong.
That is why confidence tells you nothing. Fluency is the default output, whether or not the facts exist.
Why it costs more than an obvious error
A crash gets fixed. A hallucination gets forwarded. Nobody flags it, because it looks like every good answer around it. Few readers audit a system that has been right all week.
The damage tracks the decision attached to it. A wrong restaurant tip is a shrug. A wrong drug interaction, trial match or coverage limit is not. That gap is why regulated buyers test refusal behaviour before they test accuracy.
- GroundAnswer only from fetched passages
- CiteLink the passage behind each claim
- RefuseNothing found beats a guess
- ScoreEval set gates every release
- ReviewA human signs high-stakes output
No single control is enough. Grounding cuts invention, citation makes it checkable, refusal handles the gap, and evals catch the drift.
01Can hallucination be eliminated completely?
No, and any vendor promising zero hallucination is selling you something. Grounding, citation, refusal and scored evals push the rate down and make the remainder visible. For high-stakes decisions the reliable control is still a human approving the output.
02Does RAG fix hallucination?
RAG reduces it sharply by giving the model real passages to answer from, so it has less need to invent. It introduces its own failure mode, though: an answer grounded in the wrong or outdated passage. Retrieval quality and document versioning become the thing to watch.
03Why do AI models make up citations?
A citation has a strong, predictable shape, so a model can generate a convincing one without any source behind it. Author names, years and journal titles are exactly the sort of pattern next-word prediction reproduces well. Only a lookup against a real index catches this.
Related
- how RAG grounds an answer →The main control, and its limits.
- where the embeddings live →Why the wrong passage comes back.
- rag vs fine tuning →Which approach fixes which failure.
- an assistant that cites its source →How we build and test for this.
- matching patients to oncology trials →Where a human signs off on every result.
