What is a hallucination?
Hallucination is the name given to output a generative model produces with no grounding in its inputs or its training data, invented during generation and delivered in the same register as a correct answer. The word is borrowed from psychology and is imprecise, but it stuck.
In customer support the risk concentrates on specifics: dates, amounts, eligibility windows, warranty lengths, and policy names, the parts of an answer a customer acts on immediately. The same generation mechanism produces the correct sentence and the invented one, which is why fluency reveals nothing about accuracy.
How hallucination happens
A language model predicts the next unit of text given everything before it, scoring candidates by likelihood learned from training data. Likelihood is a statistical property and truth is not, and the model holds no separate ledger of facts to check itself against, since AI tokens are the only material it manipulates. Ask for something absent from both the supplied context and the learned distribution, and the highest-scoring continuation is still generated, fluently.
Three layers decide how often that continuation reaches a customer. The retrieval layer selects the passages the model gets to read, usually through retrieval-augmented generation, and a miss there leaves the model to fill the gap from memory. The source layer is the knowledge base behind it: stale articles, two policies that contradict each other, or a procedure written without its conditions all produce confident answers built on a wrong premise. The generation layer covers prompting, sampling settings, and instructions, and a system told to always produce an answer will always produce one.
The layers compound. A weak retrieval hit plus an under-specified prompt yields the classic case: an answer citing a policy clause that was never written.
Types of hallucination
Factual: The output asserts something untrue about the world, such as a product tier, a regulatory deadline, or a fee that does not exist.
Faithfulness: The output contradicts the passage that was actually retrieved, summarizing a thirty-day return window as ninety, which is the type retrieval alone cannot prevent.
Citation: The model fabricates the supporting reference itself, inventing article IDs, ticket numbers, statute names, or links that resolve to nothing.
Reasoning: Every individual fact is correct and the chain connecting them is invalid, producing an eligibility ruling no policy supports.
Commitment: The agent invents authority it does not have, promising a refund, a callback, or an exception that no downstream system will honor.
Hallucination vs factual error vs retrieval failure vs model bias
Support teams file every wrong answer under one heading, and the heading decides who is asked to fix it. A factual error faithfully reproduces something false that was genuinely present in the source or the training data. A retrieval failure returns the wrong passage, or nothing at all, before the model writes a single word. A model bias skews outputs in a consistent direction across many unrelated inputs. A hallucination is the residue after those three are ruled out: content that had no source anywhere and appeared during generation.
What goes wrong | Where it originates | Who notices it | Detectable in logs | Call it this when | |
|---|---|---|---|---|---|
Hallucination | Content invented at write time | Generation step | Customer, usually after acting on it | Only by comparing claims to sources | The claim appears in no retrieved passage |
Factual error | True-to-source but false content | Knowledge base or training data | Subject matter expert | Yes, the source shows the same error | The wrong answer is traceable to a document |
Retrieval failure | Wrong or empty context supplied | Search and ranking layer | Engineering, via retrieval traces | Yes, the retrieved set is logged | The right passage existed and was not returned |
Model bias | Systematic skew across inputs | Training distribution | Analyst reviewing aggregates | Only in aggregate, never per answer | The pattern repeats across unrelated cases |
Look at the retrieval trace first. If the correct passage was returned and the answer still departs from it, you have a hallucination and the fix is generation-side. If the passage never arrived, or arrived carrying the error, the fix belongs to search or to content.
Why hallucination matters for customer experience
An invented answer costs more than a missing one. A customer who is told nothing escalates; a customer who is told the wrong eligibility window plans around it, discovers the truth later, and arrives at the second contact with a grievance rather than a question. Confident wrong answers also contaminate the record, because the transcript becomes evidence the customer will quote back.
The absence of any control shows up as silent failure. Nothing in a hallucinated reply looks broken, so it passes automated checks, satisfies the reader, and is discovered only when a promise fails to materialize. Teams that publish guardrails for support automation are buying detection, since prevention alone is never complete.
The tradeoff is explicit and unavoidable: every threshold that suppresses invented answers also suppresses some correct ones, so tightening a confidence score cutoff trades resolution rate for accuracy. There is no setting that gives you both.
How is hallucination measured?
No standards body sets a target hallucination figure that a customer support team is expected to hit. Academic and vendor benchmarks do exist, including TruthfulQA and HaluEval for model-level factuality and RAGAS-style faithfulness scoring for retrieval pipelines, but their task definitions and their labeling taxonomies are theirs, so their scores do not transfer to your contact reasons or your policy corpus.
The workable method is claim-level review. Sample production conversations, split each answer into its individual assertions, and have a reviewer mark every assertion as supported by a retrieved source, unsupported, or contradicting one. Report the unsupported-claim rate per sampled answer and trend it, and hold the sampling frame constant so movement means something.
Treat that measurement as part of a governance cycle rather than a one-off audit. The NIST AI Risk Management Framework defines the govern, map, measure, and manage functions that turn a sampling exercise into a repeatable control with a named owner.
How AI agents change hallucination
An AI agent shifts the consequence from words to actions. A model answering a question produces a sentence someone can dispute; an agent with tool access issues a refund, cancels a subscription, or updates an address, and an invented parameter becomes a transaction. The failure moves from the transcript into the systems of record, where reversing it is manual work.
Agent architectures also add steps where invention can enter. Multi-turn planning lets an early wrong assumption propagate through subsequent tool calls, each one inheriting the error and adding confidence to it. Tool arguments are generated the same way prose is, so an agent can hallucinate an order ID as easily as a policy.
The counterweights are structural: constrained tool schemas that reject malformed arguments, abstention paths that hand off when retrieval comes back thin, and human in the loop approval on any action with money or irreversibility attached. Practitioners comparing approaches will find tested hallucination prevention methods useful for separating claims from mechanisms.
What to look for in hallucination controls
Start with coverage. Ask which answer types are checked before delivery, since most systems verify only the ones easy to verify, leaving reasoning and commitment cases unguarded.
Integration surface decides whether verification is possible at all. A control that can read the order system, the billing system, and the policy store can check a claim against live state; one that sees only the model’s own output can check style.
Governance is the axis teams underweight. Insist on named ownership of prompts and content, an audit trail linking any production answer to the exact instruction and passages behind it, and a documented escalation route when a control fires. Security frameworks like SOC 2 Type II, ISO 27001, ISO 42001, HIPAA, and GDPR govern how that evidence is stored and who may review it.
The operational constraint is latency. Every verification pass costs time on a live conversation, and past a couple of seconds callers begin to abandon, so the number of checks you can run is bounded by patience.
Hallucination and grounding
Grounding is the discipline built specifically to contain this failure. The distinction between AI grounding and RAG matters here because retrieval is one technique for achieving grounding and grounding is the broader requirement that every claim trace to a verifiable source.
The vocabulary overlaps in practice: what this page calls a hallucination is discussed model-side as an AI hallucination, and both descriptions point at the same gap between what a system asserts and what it can substantiate.
What does hallucination mean in plain terms?
Think of a hallucination as a student who never learned to say “I don’t know.” Asked a question outside what they studied, they compose something in the shape of an answer, using the right vocabulary and the right confident cadence, because producing an answer is the only behavior they were ever rewarded for.
Without any control on this, a customer asks whether their two-year-old device is still covered, the system has no warranty document for that model, and it composes a coverage answer from what warranties usually say. The customer ships the device. It comes back uncovered.
The tradeoff is that the fix people want, a system that refuses whenever it is unsure, also refuses questions it could have answered well. Every point of accuracy bought this way is paid for in handoffs to a person.
Common hallucination mistakes
Treating it as a model problem is the first. Teams swap models expecting the rate to fall, when the invention was triggered by a knowledge base gap that any model would have filled the same way. The mechanism is missing context, and context is content work.
Trusting stated confidence is the second. Models can report high certainty on invented content because the certainty describes the token sequence rather than the fact, so a self-reported score is a weak gate on its own.
Testing only on questions with answers is the third. Evaluation sets built from documented cases never exercise the condition that causes hallucination, which is a question the corpus does not cover. Include unanswerable questions and measure whether the system abstains.
Suppressing the symptom is the fourth. Adding “do not make things up” to a prompt reduces the visible rate enough to look like a fix while leaving the retrieval gap intact, so the failures return the moment question volume shifts.
Frequently Asked Questions
Why does AI hallucinate?
AI hallucinates because generative models are optimized to produce likely text, and likelihood correlates with truth only where training data and retrieved context actually cover the question. When coverage is thin, the model still generates its best-scoring continuation, and no internal mechanism flags the difference between recalling something and composing it.
What is the difference between a hallucination and a factual error?
A hallucination is content with no source behind it, invented while the answer was being written. A factual error is content that faithfully reflects a source which happens to be wrong or outdated. The distinction determines the fix: hallucination is addressed through grounding and abstention, factual error through correcting the underlying document.
Hallucination vs confabulation: which term is more accurate?
Confabulation is the more accurate borrowing, since it describes filling a memory gap with plausible invented detail while genuinely believing it, which is closer to what happens mechanically. Hallucination has become the settled industry term regardless, and both describe the same behavior, so precision here is a vocabulary preference rather than a technical one.
Can hallucination be eliminated completely?
Hallucination cannot be eliminated while a system generates free text, because generation and invention share one mechanism. It can be constrained sharply: retrieval that returns the right passage, prompts that permit refusal, schema validation on tool arguments, and escalation when evidence is thin all reduce how often invented content reaches a customer.
What is an example of a hallucination in customer support?
A common example is invented policy detail. A customer asks about a return deadline for a product line the help center never documented, and the agent produces a specific window drawn from how return policies usually read. The number is confident, formatted correctly, and unsupported by anything the company has published.
How do you detect hallucinations before a customer sees one?
Detecting hallucinations before delivery requires comparing each claim in a draft answer against the passages actually retrieved and against live system state such as order or account records. Claims with no matching evidence get suppressed, rewritten, or escalated. Sampled human review of production transcripts catches the categories automated comparison misses, particularly invalid reasoning.