Why your AI answers wrong: 9 knowledge base failure modes

Table of Contents

• Wrong Answers Are a Symptom, Not a Diagnosis

• Failure Mode 1: The Missing Passage

• Failure Mode 2: The Contradiction

• Failure Mode 3: The Stale Absolute

• Failure Mode 4: The Orphaned Chunk

• Failure Mode 5: The Near Miss

• Failure Mode 6: The Buried Condition

• Failure Mode 7: The Descriptive Dead End

• Failure Mode 8: The Internal Note

• Failure Mode 9: The Unanswerable Question

• Telling Them Apart From the Transcript

• The Triage Table

• Instrumenting for Diagnosis Instead of Guesswork

• Final Verdict: Fix the Document, Not the Prompt

TL;DR. Nine distinct content failures produce a wrong AI support answer, and they have different fixes. Missing passages make the agent improvise. Contradictions make it confidently pick a side. Stale absolutes state retired facts as current. Orphaned chunks lose the procedure they belonged to. Near misses retrieve the adjacent plan or region. Buried conditions answer the default case for a customer in the exception. Descriptive dead ends are correct and useless. Internal notes leak. Unanswerable questions need account data, not documents. Diagnose from the transcript signature before you change anything, because the most common mistake is tuning the prompt against a problem that lives in a document. Fini surfaces the retrieved passages alongside each answer for exactly this reason.

Wrong Answers Are a Symptom, Not a Diagnosis

The support lead brings a screenshot to the standup. The agent told a customer the refund window was fourteen days when it is thirty. Someone says the model is hallucinating. Someone else suggests adding a line to the system prompt saying the refund window is thirty days.

Both responses are wrong, and the second one is actively harmful, because it patches a single symptom while the cause keeps producing new ones. In this case the cause was almost certainly a legacy regional policy page that nobody remembered existed, and the same page is going to produce a wrong answer about exchange eligibility next week.

A retrieval-grounded agent composes answers from passages it finds in your corpus. When the answer is wrong, the interesting question is which passage it used and why it was the best available match. There are nine common answers to that question. They look similar from the outside, and they need entirely different repairs.

The diagnostic discipline here is worth building deliberately, because the default failure loop in most teams is to escalate every wrong answer to prompt engineering. That loop produces a system prompt of four thousand words full of policy exceptions, which is both unmaintainable and a worse retrieval corpus than the documents it is trying to override.

Failure Mode 1: The Missing Passage

Symptom. A fluent, plausible, entirely invented answer. Often about a feature that sounds like something you would have, or a policy that resembles an industry norm. The tone is confident and the specifics are wrong in ways that are hard to spot without domain knowledge.

Why it happens. Retrieval returned nothing above the relevance floor, or returned passages that are only topically adjacent. What happens next depends on the platform. Some escalate. Some say they do not know. Others compose an answer from whatever came back plus the model's general knowledge of how companies like yours usually work, and that last behavior is where invented answers come from.

How to confirm. Look at the retrieved passages for that turn. If the top result has a low relevance score, or is about a different topic entirely, this is a coverage gap rather than a model failure. If your platform does not show you the retrieved passages, that is a tooling problem worth fixing before anything else, because you cannot diagnose the remaining eight failure modes without it.

The fix. Write the article. But fix the behavior first, because coverage is never complete: configure the agent to say it does not know and hand off rather than improvise. A "let me get someone who can confirm that" is a better outcome than a fluent invention, and it converts a silent failure into a visible one you can count. Platform differences in this exact behavior are covered in comparisons of AI support platforms with hallucination guardrails.

Failure Mode 2: The Contradiction

Symptom. The answer is wrong in a specific, quotable, numeric way. Fourteen days instead of thirty. A twenty-five dollar fee that was removed. The answer is also internally consistent and confident, because the passage it came from is a real document that says exactly that.

Why it happens. Two or more passages in the corpus make incompatible claims. Retrieval picked one. There is nothing in a passage that flags it as contradicted elsewhere, so the agent has no way to know.

How to confirm. Search the corpus for every passage mentioning the disputed value. If you find two, you have your answer in about ninety seconds. Numeric contradictions are the easiest failure mode to confirm and the most urgent to fix.

The fix. Pick a canonical article, update it, and delete or redirect the others. Do not simply update the second article to match, because you now have two articles that will drift apart again at the next policy change. Consolidation is the durable fix; agreement is temporary.

Contradictions concentrate in predictable places: policies that changed within the last two years, regionally duplicated content, and anywhere a team wrote their own version rather than editing the shared one. Detection tooling for this pattern is compared in guides on AI knowledge bases that catch conflicts.

Failure Mode 3: The Stale Absolute

Symptom. The agent tells a customer they cannot do something they can now do. "Our platform does not currently support single sign-on." "Bulk export is not available." Sometimes the reverse: promising a feature that was deprecated.

Why it happens. Someone wrote a categorical statement in a document, the world changed, and the document did not. Absolutes are especially damaging because they are unhedged. A page that said "SSO is on our roadmap" ages gracefully. A page that says "SSO is not supported" becomes a lie the moment it ships.

How to confirm. Cross-reference against your product change log rather than against modification dates. A last-modified date tells you when someone touched the file, not whether it is true. Take every change shipped in the last four quarters and ask which articles should have moved. The gaps are your stale absolutes, and they cluster where the product moved fastest.

The fix. Repair the specific pages, then reduce future exposure by changing how absolutes get written. "Not supported" becomes "not supported as of the last update to this page, check the release notes." That is a weaker sentence and a much more robust one. The systemic fix is a release checklist item: every shipped change names the articles it invalidates. Tooling that flags stale and conflicting content automates the detection, but the release-time habit is what keeps the backlog from regrowing.

Failure Mode 4: The Orphaned Chunk

Symptom. The answer contains real steps from your documentation in an order that does not make sense, or steps that belong to a different procedure than the one the customer asked about. Everything in it is quotable from your corpus and the assembly is wrong.

Why it happens. Retrieval systems split documents into chunks. A chunk from the middle of a long page arrives without the heading that gave it meaning. A chunk that reads "then enter the code from your authenticator app and click Verify" could belong to login, to enabling two-factor, or to recovering an account. The retriever cannot tell, and neither can the model.

How to confirm. Find the source document for the retrieved chunk and look at where the chunk boundaries fell. If the chunk does not name what procedure it belongs to, you have found it. This failure mode correlates almost perfectly with document length: pages over about two thousand words covering multiple procedures are where it lives.

The fix. Split long pages by procedure so each document covers one task end to end. Use descriptive headings, since most pipelines prepend the heading to the chunk and a heading like "How to reset a lost authenticator device" carries the context the body assumes. Where splitting is not practical, repeat the identifying context in each section rather than relying on a page title three thousand words up.

Tables deserve specific attention. A table row separated from its header produces a chunk reading "Yes, 48 hours, Enterprise only" with no indication of the subject. Where a table drives frequent questions, write a prose summary beneath it that survives chunking on its own.

Failure Mode 5: The Near Miss

Symptom. A correct answer to a question the customer did not ask. The Business plan limit given to an Enterprise customer. The EU return policy given to a US customer. The mobile app procedure given to someone on the web.

Why it happens. Semantic retrieval matches on similarity, and your Business plan page and Enterprise plan page are extremely similar documents. They differ in the specific numbers, which is exactly the part embeddings represent least well. If the customer's question does not include the distinguishing term, both pages match and the retriever picks by a margin that is close to noise.

How to confirm. Check whether a sibling document exists that would have been correct. If the answer is verbatim right for a neighboring plan, region, or platform, this is a near miss rather than a contradiction. The distinction matters: contradictions need consolidation, near misses need disambiguation.

The fix. Three layers, in order of durability.

Make the distinguishing attribute prominent in the document rather than implicit. A page that says "Enterprise plan" once in the title and never again gives retrieval one weak signal. Repeat it in headings and in the sentences that carry the numbers.

Pass what you know as context. If the customer is authenticated and you know their plan, region, and platform, that belongs in the retrieval context so the system is not guessing from phrasing.

Ask when it matters. For unauthenticated conversations where the answer differs materially by segment, one clarifying question is far better than a confident wrong number. This is a configuration decision, and the right threshold depends on how much the answers diverge.

Failure Mode 6: The Buried Condition

Symptom. An answer that is right for most customers and wrong for this one. The refund policy is thirty days, and it is, except for sale items, which is stated in a sentence near the bottom of a different page.

Why it happens. Exceptions and conditions are written where the person documenting them was thinking, which is rarely where the retriever looks. The main policy page states the general rule cleanly and the exception lives in a footnote, an FAQ entry, or a separate page for the exceptional case. Retrieval matches the general question to the general page, correctly, and the condition never enters the context.

How to confirm. Ask whether a human agent would have caught it. If your best agent would have said "wait, was this a sale item?", the knowledge existed and was not retrievable. This is the failure mode most likely to be misread as an agent quality problem, because the retrieved passage supports the answer perfectly. The passage is just incomplete.

The fix. Move conditions next to the rule they modify. The refund policy page should state the window and immediately list every category the window does not apply to, even though this makes the page longer and less elegant. Elegance in a policy page is a retrieval liability.

Where the exception list is long, invert the structure: state the rule, then a compact table of exceptions with the condition and the alternate treatment. Tables chunk badly, so add a one-line prose summary of the exceptions beneath the table. Redundancy inside a document is cheap; a missing condition is expensive.

Failure Mode 7: The Descriptive Dead End

Symptom. The answer is accurate, on topic, well written, and the customer opens a ticket anyway. Nothing in the transcript looks like a failure. Deflection metrics look healthy while resolution does not move.

Why it happens. The content describes rather than resolves. Most help centers were written to explain a product to someone browsing it, not to unstick someone who is stuck. The agent faithfully reproduces "Team plans support single sign-on, administrators can configure SAML from security settings" to a customer who is looking at an error message about a certificate.

How to confirm. Read the answer as if you had the problem. If your next move would be to ask a follow-up question, the source article is descriptive. Then check the pattern at scale: look for intents with high answer rates and high subsequent ticket creation. That combination is this failure mode, and it is invisible on a deflection dashboard.

The fix. Rewrite the top intents to carry the resolution: the actual steps, the prerequisites, the two most common error states, and what to do when the standard path does not apply. Source the material from your macro library and from how your strongest agents actually resolve the ticket, not from the product documentation.

This is the highest-effort fix on the list and usually the highest return, because it moves the metric everyone actually cares about. It is also the one most likely to be skipped, because the failing answers do not look like failures.

Failure Mode 8: The Internal Note

Symptom. The agent tells a customer something true and internal. An escalation threshold, a discount ceiling, a note about which accounts get priority handling, the existence of a workaround the support team was told not to publicize.

Why it happens. Internal content sits in the same retrievable corpus as customer-facing content, either because it was never labeled or because the labels exist and retrieval does not enforce them. The most common shape is not a separate internal document but a customer-facing page with an internal paragraph at the bottom, often a callout reading "Agents: you can approve up to two hundred dollars without a manager."

How to confirm. Search the customer-facing corpus for the vocabulary of internal instruction: "agents should", "do not tell the customer", "escalate to", "manager approval". This takes ten minutes and the results are frequently uncomfortable.

The fix. Label every article with an audience, verify retrieval enforces the label, and split mixed articles rather than trying to instruct the model around them. A prompt saying "never reveal internal information" is not an access control, because the model cannot reliably tell which of the passages in its context is internal.

There is a positive side to this. Agent-facing content is often the best-written material you have, and the right configuration exposes it to an agent-assist deployment while keeping it out of the customer-facing one. That distinction, covered in guides on agent-facing knowledge bases, is a capability rather than merely a restriction.

Failure Mode 9: The Unanswerable Question

Symptom. The agent produces a generic answer to a specific question. "Why was my payment declined" gets a list of reasons payments are sometimes declined. "Where is my order" gets an explanation of shipping timelines.

Why it happens. There is no document that answers this question, and there cannot be. The answer depends on the state of an account, an order, or a transaction, which lives in a system rather than in a page. The agent does what it can with the corpus, and the corpus can only offer the general case.

How to confirm. Ask whether any document could have answered correctly. If the answer requires a lookup, this is not a content failure and no amount of writing will fix it.

The fix. Integration, not documentation. The agent needs read access to the order, subscription, or payment record. This is a different project with a different cost profile, and the important thing at diagnosis time is to classify these correctly and stop counting them against content quality.

Until the integration exists, the correct behavior is a fast, clean handoff that carries the context so the customer does not repeat themselves. A generic answer to a specific question is the worst available outcome: it burns a turn, tells the customer nothing, and signals that the agent did not understand. Handoff quality is its own discipline, covered in guides on handoff quality and context preservation.

Telling Them Apart From the Transcript

In practice you diagnose from three artifacts: the customer's question, the agent's answer, and the passages retrieved. With all three, most cases resolve in under two minutes using four questions in order.

Was anything relevant retrieved? If no, you are in failure mode 1 or 9, and the split is whether a document could exist. If the question needs account state, it is 9.

Is the retrieved passage true? If the passage itself is wrong or outdated, you are in 2 or 3. Search for a competing passage: if one exists, it is a contradiction; if not, it is a stale absolute.

Is the passage true but wrong for this customer? That is 5 or 6. If a sibling document would have been correct, it is a near miss. If the correct answer required a condition documented elsewhere, it is a buried condition.

Is the passage true, correct, and still unhelpful? That is 4, 7, or 8. Garbled or misassembled steps mean an orphaned chunk. Accurate but non-resolving means a descriptive dead end. Internal content means an audience gating failure.

The one step that makes this fast is having the retrieved passages next to the answer. If your platform hides them, every diagnosis becomes a guess, and guessing pushes teams toward prompt patches because that is the only lever they can see.

The Triage Table

Two things stand out from the effort column. Four of the nine are low effort, and three of those four (contradictions, buried conditions, internal notes) produce the most damaging category of error, which is confident and specific and wrong. Start there. It is usually a week of work and it removes the failures that generate escalations.

The two high-effort fixes are the ones that move containment rather than accuracy. Descriptive dead ends and unanswerable questions are why an agent with an excellent accuracy rate can still contain a third of what the business case assumed.

Instrumenting for Diagnosis Instead of Guesswork

Diagnosing case by case works at ten wrong answers a week and stops working at two hundred. Four instruments make it scale.

Log retrieved passages with every answer. This is the prerequisite for everything else. Store the passage identifiers, the relevance scores, and the final answer together so a wrong answer can be traced to a document rather than argued about.

Alert on low-confidence answers rather than only on escalations. The dangerous population is answers the system produced with weak retrieval support, and by definition they did not escalate. Sample them weekly. This is where failure modes 1 and 5 hide.

Track answered-then-ticketed as a first-class metric. For every intent, measure how often a customer who received an answer opened a ticket within twenty-four hours. High rates on an intent with high answer rates is the signature of failure mode 7, and it appears nowhere on a standard deflection rate dashboard.

Cluster wrong answers by source document, not by intent. One bad document usually produces many wrong answers across several intents, so a per-intent view splits one root cause into six symptoms. Sorting by source document collapses them and makes the actual work list obvious. It is common to find that four documents account for a third of all wrong answers.

Fini exposes the retrieved passages alongside every answer and clusters failures back to source documents for this reason: the difference between a team that fixes content and a team that tunes prompts forever is almost entirely whether they can see which passage produced the answer.

Final Verdict: Fix the Document, Not the Prompt

The instinct when an AI agent answers wrong is to add an instruction. It feels like the fast fix, it is visible, and it works for exactly the case you tested. Then the same underlying document produces a different wrong answer next week, and the instruction list grows into an unmaintainable second knowledge base that the model reads before your actual one.

Almost every wrong answer traces to a document. It is missing, contradicted, stale, split badly, too similar to its neighbor, missing its exception, describing instead of resolving, internal, or absent for good reason because the answer lives in a database. Nine causes, four of them cheap to fix, and all nine diagnosable from the transcript plus the retrieved passages.

Start by logging what was retrieved. Then spend a week on contradictions, buried conditions, and internal notes, which is the highest return available and removes the errors that generate escalations. Then decide whether the remaining gap is a coverage problem or an integration problem, because those are different projects and conflating them is how automation programs stall in month four.

To see how failures trace back to source documents in your own corpus, talk to our team.

Frequently Asked Questions

Why does my AI support agent make up answers?

Almost always because retrieval found nothing above the relevance floor and the platform is configured to compose an answer anyway. The model fills the gap with general knowledge of how companies like yours usually operate, which produces something fluent and plausible and wrong. The fix has two parts: write the missing content, and change the fallback behavior so the agent says it does not know and hands off. The second part matters more, because coverage is never complete and you want gaps to fail visibly rather than silently.

What is the difference between a contradiction and a near miss?

A contradiction is two passages in your corpus making incompatible claims about the same thing, so whichever one retrieval picks, the other says it is wrong. A near miss is two passages that are both correct but describe different segments, such as the Business plan and Enterprise plan pages, where retrieval picked the wrong one because the question did not contain the distinguishing term. Contradictions are fixed by consolidating to a single canonical article. Near misses are fixed by making the distinguishing attribute prominent and passing what you know about the customer into retrieval.

How do I know if a wrong answer is a content problem or a model problem?

Look at the passages that were retrieved for that turn. If a relevant, accurate passage was retrieved and the answer still contradicts it, that points at the model or the prompt. In practice this is the minority of cases. Far more often the retrieved passage is missing, outdated, internally contradicted, or correct for a different customer segment, all of which are content problems. If your platform does not expose the retrieved passages, fix that first, because without it every diagnosis is a guess.

Which knowledge base failure mode should I fix first?

Contradictions, buried conditions, and internal notes, which are all low effort and produce the most damaging errors: confident, specific, and wrong. Numeric contradictions in your top intents can usually be found and consolidated in a few days. Buried conditions are fixed by moving exceptions next to the rule they modify. Internal notes are found by searching the customer-facing corpus for phrases like "agents should" and "do not tell the customer". Together this is roughly a week of work and it removes the failures most likely to generate escalations.

Why does my AI answer correctly but customers still open tickets?

This is the descriptive dead end. The content is accurate but explains the product rather than resolving the problem, so the customer reads a true statement and remains stuck. It is invisible on a deflection dashboard because the answer rate looks healthy. Detect it by tracking, per intent, how often a customer who received an answer opens a ticket within twenty-four hours. Fix it by rewriting the highest-volume intents to include the actual steps, prerequisites, and the two most common error states, sourced from your macro library rather than from product documentation.

Can prompt engineering fix knowledge base problems?

Only narrowly, and it degrades over time. A system-prompt instruction can override one specific wrong fact, but the underlying document keeps producing new variants of the error, and each patch adds to a growing instruction list that eventually competes with your actual corpus for the model's attention. Prompt instructions also cannot enforce access control, because the model cannot reliably distinguish internal from customer-facing passages once both are in context. Treat prompt patches as a temporary hold while the document fix is scheduled.

How do orphaned chunks break AI answers?

Retrieval splits documents into chunks, and a chunk taken from the middle of a long page arrives without the heading that gave it meaning. A step reading "enter the code from your authenticator app and click Verify" might belong to login, to enabling two-factor, or to account recovery, and nothing in the chunk says which. The agent then assembles real steps into a procedure that does not exist. This correlates strongly with document length, so the fix is splitting pages over about two thousand words into one document per procedure and using headings that name the task.

What should the AI do when no document can answer the question?

Hand off with context, quickly. Questions like "why was my payment declined" depend on account state rather than documentation, so no article can answer them and the worst outcome is a generic response about reasons payments are sometimes declined. That burns a turn and signals the agent did not understand. The durable fix is integration, giving the agent read access to the relevant record, which is a separate project from content work. Classify these cases separately so they stop being counted against knowledge base quality.