Hallucination detection

What is hallucination detection?

Hallucination detection is the set of automated checks that decide whether an AI agent's drafted answer is supported by the sources it retrieved, run before or immediately after that answer reaches the customer. It scores each claim, flags the unsupported span, and triggers a defined response.

Detection is a separate concern from generation: the model that wrote the sentence is a poor judge of it, so most production systems run the check as a second pass that sees the answer, the retrieved passages, and the account record the answer claimed to read.

How hallucination detection works

A detector runs on the same turn as the answer, in four stages: claim extraction, source alignment, scoring, and action.

Claim extraction splits the drafted reply into individually checkable assertions: an amount, a date, a policy condition, an order status. Fluent text hides three or four claims inside one sentence, so the granularity of the split decides how much of the answer is really inspected.

Source alignment matches each claim back to the passages the retriever returned from the knowledge base and to any structured record the agent called during the turn. A claim with no matching span is unsupported by construction, and that is the signal separating a hallucination from an answer that is merely clumsy.

Scoring assigns a support verdict per claim. Usually a second large language model is prompted as a strict entailment checker; where the claim is a number that exists in a system of record, deterministic comparison does the job faster and more reliably.

Action is the stage teams underbuild. A verdict with no policy attached changes nothing about what the customer receives, so the pipeline has to rewrite with citations, suppress, hedge, or hand off.

Types of hallucination detection

• Grounding and entailment checks: A second model judges whether each claim follows from the retrieved passages, span by span, though it inherits every blind spot the retriever had.

• Deterministic field validation: Amounts, dates, and statuses in the draft are compared directly against the system of record that owns them, which catches the costliest errors cheaply.

• Self-consistency sampling: The same question is answered several times and divergence across samples is read as a confidence signal, at real cost in latency and tokens.

• Uncertainty signals: Token probabilities and retrieval scores act as a proxy for thin ground, useful as a cheap pre-filter ahead of a heavier check.

• Policy and schema validators: Deterministic rules reject answers that assert commitments the agent may never make, such as a refund outside the published window.

Hallucination detection vs guardrails vs AI evals vs human QA review

All four sit inside the same accuracy program and often arrive as one budget line, which is where teams start using the names interchangeably. Guardrails apply fixed policy to categories of output, blocking a shape of reply whatever its grounding. AI evals score a frozen set of cases offline, reporting what a change did before it ships. Human QA review rates a sample of already-closed conversations with full hindsight and coaching intent. Hallucination detection asks one narrow question of every live answer: does a source support this specific claim.

If you already run AI evals and customers still report answers that were never true, the gap is at runtime and detection is what closes it. If the bad answers all fall into categories you can name in advance, guardrails are the cheaper and more predictable fix.

Why hallucination detection matters for customer experience

A wrong answer looks identical to a right one on the way out: same tone, same structure, same confidence. With no detector in the path, the first reviewer is the customer, and by then they have quoted a refund window that does not exist or cancelled a plan on advice the policy never gave. The cost lands twice, once in the reopened ticket and once in the trust the next answer no longer earns.

The tradeoff is friction. Tighten the support threshold and the detector suppresses correct answers alongside invented ones, which sends more conversations into the escalation queue and raises handling time for the people behind it. Teams that treat AI guardrails as a dial tuned per intent keep that cost visible instead of discovering it in a backlog.

How is hallucination detection measured?

Faithfulness scoring inside evaluation suites like RAGAS gives the underlying task a public yardstick: each claim in a generated answer is scored against the passages retrieved for it, and the suite reports how much of the answer was grounded. Those figures describe that suite's claim taxonomy and its test corpora, so they carry no information about how your detector behaves on refund questions asked by your customers. Nobody publishes a target flag rate a support team is expected to hit, and no standards body defines one, which leaves your own baseline as the only meaningful reference.

The method is a labeled sample. Take a batch of live answers, have people mark each claim supported or unsupported using the same sources the agent had, then measure how many flags were genuine errors and how many genuine errors were flagged. The measure function of the NIST AI Risk Management Framework describes this cycle: define the metric, test against representative data, and keep testing once the system is live.

How AI agents change hallucination detection

An agent that only writes text produces a single artifact to verify. Agentic AI produces several: a plan, a sequence of tool calls, arguments filled from the conversation, and a closing summary of what it did. An invented claim now surfaces as a fabricated order ID passed to a refund endpoint, or a confident summary of an action that silently failed. Detection therefore has to move upstream of the reply, checking each call's arguments against the record before execution and checking the summary against the responses that came back.

The consequence is a step-level gate. Multi-step plans compound: a wrong premise in step one is carried as established fact through every later step, so a check that inspects only the final message is inspecting the smallest part of the risk. Teams building this pair per-step verification with hard policy limits, an approach set out in this comparison of accuracy and hallucination guardrails.

Implementing hallucination detection

Start with the latency budget, because it eliminates methods before anything else does. A voice turn leaves a couple of hundred milliseconds of headroom, which takes repeated sampling off the table; an email draft can absorb a full second pass through a second model.

Then decide what the checker is allowed to see. A detector with access to the retrieved passages can judge grounding. One that also reads the order and account records can verify the numbers themselves, which is the higher ceiling and the harder integration.

Write the action policy before tuning any threshold: rewrite with citations, suppress and reroute, or send with a hedge. Each threshold needs a named owner reviewing it on a schedule, since knowledge content shifts weekly and a threshold tuned in March describes March.

Two frameworks bite here. SOC 2 Type II governs how long flagged spans are retained, and those spans are copies of customer data sitting in a review queue. An AI management system standard such as ISO/IEC 42001 forces the piece teams skip: documented ownership of thresholds plus evidence someone acted on the flags. The constraint that ends most programs is queue capacity, because a detector producing flags nobody reads is an expensive logging system.

Hallucination detection and release safety

Detection produces the signal that makes a careful rollout possible. A canary deployment exposes a new prompt, model, or retriever to a slice of live traffic and watches guard metrics, and unsupported-claim rate is one of the few guard metrics that moves fast enough to catch a bad retrieval change within the hour.

That signal is only trustworthy if the detector itself has been checked, which is the job of a golden dataset of human-verified answers including known-bad ones: it tells you what share of real errors your checker caught before you let it gate live traffic.

What does hallucination detection mean in plain terms?

Think of it as the fact-checker sitting between a very fast writer and the printing press, reading each sentence against the file the writer was handed and marking anything that is not in that file.

Remove the fact-checker and nothing looks different. The page still prints, the sentences still scan, and the error only becomes visible when a reader acts on it and something goes wrong at their end.

The tradeoff is that the fact-checker is slower than the writer and sometimes cautious in the wrong direction, holding a true sentence because the file was thin that day. Every team decides how much delay and how much unnecessary doubt it will accept, and the honest answer depends entirely on what one wrong sentence costs when it reaches a customer.

Common hallucination detection mistakes

Grading with the same model, prompt, and context that produced the answer. The checker re-derives the claim along the same path and confirms itself, so the errors most worth catching are exactly the ones it is structurally blind to.

Shipping verdicts with no action attached. A support score that writes only to a dashboard changes nothing about what the customer receives, and by the second month the dashboard has stopped being opened.

Tuning the threshold once. Content, prompts, and models all move underneath a fixed threshold, and its false-flag rate drifts quietly until either customers hit errors or agents learn to dismiss every flag on sight.

Reporting flag volume as the headline number. A detector blind to a whole class of error, unsupported policy conditions for instance, produces a calm chart, and calm charts are why nobody goes back to the labeled sample that would expose the miss.