Two numbers that look similar can hide a 30-point gap in actual performance: deflection rate and resolution rate. Vendors know which one looks better in a demo, and most evaluations never push past it. If your shortlist process relies on what vendors claim instead of what they can prove on your own tickets, this guide is worth reading before your next call.

TLDR:

• Per-resolution pricing is the only model that ties vendor revenue to actual outcomes. Published rates range from $0.50 to $2.00 per resolution.

Why most AI vendor evaluations fail before they start

Vendor demos are built to impress, not to predict. Most show the best-case ticket on a clean knowledge base, with a question the AI has seen before. What they don't show is month three, when the knowledge base has drifted and the edge cases have multiplied.

The pressure to pick something is real. 66% of service orgs use AI agents, up from 39% in 2024. That adoption spike means most buyers are making decisions on compressed timelines, which is exactly when feature checklists replace performance tests. Comparing AI customer service agents side by side before timelines compress is the only way to avoid that trap.

Vendors get scored on what they claim, not what they resolve. Choosing wrong means a second evaluation in six months, a failed rollout your CEO remembers, and a support team that lost trust in AI before the right vendor had a chance.

Resolution rate vs. deflection rate

Deflection rate counts tickets the AI touched without a human agent. Resolution rate counts tickets where the customer's problem was actually solved. Those two numbers can diverge by 30 percentage points, and vendors know which one looks better in a demo. The gap between deflection rate vs. true resolution rate is where most vendor pitches hide their real numbers.

A deflected ticket that sends a customer to a help article they already read is not a resolution. As IrisAgent's analysis of the metric puts it, "a deflected ticket and a resolved ticket are not the same outcome." High deflection with flat or falling CSAT is the clearest signal your vendor is optimizing for the wrong number.

Ask every vendor on your shortlist one question: what is your resolution rate, defined as tickets closed end to end with no human handover and no re-contact within 48 hours? If they answer with deflection or containment figures instead, that answer tells you everything.

The five dimensions of a rigorous evaluation

A good evaluation framework doesn't change by vendor. These five dimensions apply regardless of who's on your shortlist.

No vendor should get a pass for skipping any row. A system that scores well on accuracy but fails on escalation quality will still generate complaints. Testing which vendors actually prevent support AI hallucinations should happen before that system touches production traffic. One that passes safety tests in a demo but has no audit export will fail a compliance review in production.

What to ask about resolution numbers

Resolution statistics are easy to inflate. A vendor can report 85% resolution by defining resolution as "no reply within 24 hours," drawing from a ticket mix of simple FAQs, or pulling numbers from a cherry-picked pilot cohort instead of full production traffic.

Three questions cut through this quickly:

• What is your re-contact rate? A resolved ticket that generates a follow-up within 48 hours was not resolved.

• What was the ticket mix? Resolution rates on FAQ-only traffic don't predict performance on billing disputes or account issues.

• Is this a cohort-wide number or a selected subset? Ask for the methodology in writing, not as a verbal summary on a call.

The definition of resolution in the contract matters as much as the number itself. Confirm it means the customer's issue was closed end to end with no human handover, not simply that the AI responded and the customer stopped replying.

Agentic actions vs. information retrieval

Most AI support vendors retrieve information. A customer asks why their payment failed; the system finds the relevant help article and surfaces it. That is retrieval. It is useful, but it is not resolution.

Agentic action means the system queries Stripe, finds the duplicate charge, reverses it, and closes the ticket. The customer got their money back. No human touched it.

The difference matters most on the tickets that cost the most: billing disputes, account updates, status checks that require a live data pull. A retrieval system handles FAQ volume. An agentic system handles the work.

To test for this, skip the scripted demo. Send five tickets that require a real action: a refund, an account flag, a policy lookup tied to a specific account state. A retrieval vendor will return a help article or escalate. An agentic vendor will connect to your billing system, pull the record, and act.

One clarifying question to ask vendors: "When the agent takes an action in a customer's account, what does the audit trail capture?" A vendor with genuine action-taking capability will have a specific answer, logging the input, the system queried, the action taken, and the outcome. A retrieval vendor will not have an answer, because there is nothing to log.

Knowledge management and self-maintenance

Knowledge bases degrade quietly. Policies change, edge cases accumulate, and the AI starts answering last quarter's rules. Most vendors put that maintenance burden back on your ops team.

Ask vendors three things: how does the system detect when its knowledge is wrong or incomplete? What happens when two articles contradict each other? And what does your ops team actually have to do each week to keep it current?

The answers divide vendors fast. Manual systems require a human to spot gaps and write fixes. Self-maintaining systems detect gaps from real ticket outcomes, flag conflicts, and surface drafts for review.

Fini answers all three. Knowledge Atlas runs a nightly pipeline that identifies knowledge gaps from escalated conversations and drafts articles for human review before publishing. Conflict detection flags duplicate or contradictory instructions and outputs a reconciliation dashboard. The ops burden drops to reviewing AI-flagged edge cases, not rebuilding the knowledge base from scratch each month.

AI-human handoff and escalation quality

Escalation is where most AI support deployments quietly fail. The AI can't answer, hands off to a human, and the human starts from scratch reading the full conversation thread. That failure is a design question, and it should be part of your evaluation before you sign.

When scoring handoff quality, test three things:

• Does the escalated ticket include a structured summary, or just the raw conversation log?

• Can a human agent act immediately, or do they need to re-read the full thread first?

• Does the system log why it escalated, including the confidence score and the policy check that triggered the handoff?

That third point matters in compliance-bound industries. If your AI escalates a billing dispute with no record of why, that gap will surface in an audit.

The second architecture question is the return path. When a human resolves something the AI missed, does that resolution feed back into the system? How different platforms define bot-to-human escalation rules determines whether that return path exists at all. Or does the same question escalate again next week?

Fini's escalation logic runs at three confidence levels: high confidence resolves automatically, mid confidence drafts a response for agent review, and low confidence or policy-sensitive queries escalate with full context and an AI-generated summary attached. The human agent receives a briefing, not a thread. Every human resolution feeds the nightly Knowledge Atlas pipeline, so the same gap doesn't repeat.

Test this in your pilot by sending ten tickets that should escalate and reviewing what the human agent receives on the other end.

Compliance and security requirements

Compliance review belongs at the start of vendor evaluation, not after a demo has already won you over. Use a compliance cert checklist for AI support to set the baseline before any vendor presents. In 2026, the bar is higher than it was two years ago. COSO published new guidance on AI and internal controls in February 2026, the EU AI Act reached full enforcement in August 2026, and the SEC announced a dedicated SOX enforcement group in March 2026. Together, these make a defensible AI audit trail a regulatory requirement, not a best practice.

At minimum, require SOC 2 Type II, ISO 27001, data residency documentation, and a full decision audit trail. For fintech, add PCI DSS. For healthcare, HIPAA-compliant and BAA-eligible are both required, not one or the other.

One distinction matters here: a vendor's internal data security posture is separate from whether the agent itself acts in compliance with the regulations governing your business. SOC 2 attests to how a vendor handles your data internally. It says nothing about whether the agent understands forbearance rules, fair debt collection requirements, or claims handling obligations. Ask for both, separately. A structured comparison of enterprise AI customer support platforms covers both dimensions in parallel so neither gets skipped.

Fini ships SOC 2 Type II · PCI DSS Level 1 · ISO 27001 · GDPR · HIPAA-compliant · BAA-eligible · CCPA by default, with every agent decision logged and exportable for audit.

Pricing models and total cost of ownership

AI support vendor pricing in 2026 runs across five models: flat-included, per-resolution, per-conversation, hybrid seat-plus-meter, and expiring credits. per-resolution rates range from $0.50 to $2.00, a 4x gap on the same unit of value. The starting price on a vendor's homepage is almost never what a growing team pays at month 12.

The model matters more than the number. A full breakdown of AI customer support pricing, TCO, and ROI shows how per-seat pricing charges regardless of whether the AI resolves anything. Per-conversation pricing runs the meter on every contact, including ones that escalate. Only per-resolution pricing ties vendor revenue to actual outcomes.

To model true 12-month cost, you need three inputs: your current monthly ticket volume, your expected growth rate, and the vendor's definition of a billable event.

Contract terms worth reviewing before you sign:

• How is "resolution" defined? Confirm it means closed end to end with no human handover.

• Is there a penalty rate for overages, or does the same per-resolution price apply?

• Does unused allowance roll forward, or does it expire at month-end?

Fini prices per resolved ticket, with no per-seat fees and no penalty overage rate. Escalations to your team are free. Plans run $0.89 on Growth, $0.69 on Scale, and $0.49 on Enterprise, with overages billing at the same flat rate.

How to run a pilot that predicts production performance

Send real production tickets, including the messy ones: billing disputes, account-state questions, regulatory edge cases. A vendor who asks to exclude billing disputes from the pilot mix is hiding their hardest failure mode.

Define success metrics in writing before the pilot begins: resolution rate, re-contact rate, accuracy against known ground truth. If the vendor won't commit those targets to paper before the pilot runs, they won't hit them after.

Two structural requirements matter here. First, agree on minimum ticket volume and duration upfront. Second, test graceful failure deliberately by including tickets the AI should escalate, then score what it actually does.

Fini's Zero-Pay Guarantee formalizes this on the Enterprise plan: 90% resolution in 90 days on live traffic, or you pay $0.

Building the buying committee and evaluation scorecard

A vendor decision made by one person is a vendor decision made on one set of priorities. The CX leader cares about resolution numbers. The CTO cares about audit trails and data residency. The Ops Lead cares about what week six actually looks like. Each dimension needs an owner, or the evaluation collapses into whoever presents the demo best.

A scorecard keeps that from happening. Reviewing the best AI customer service software options before weighting dimensions anchors each score to a real capability. Weight each dimension before vendors are assessed, not after.

One rule: weights are set before any vendor presents. Changing them after a demo to favor a preferred vendor is how evaluations get captured.

Each owner scores their dimensions independently, then the committee meets once to compare. If the CTO flags a compliance gap that the CX leader ranked as acceptable, that is a conversation worth having before the contract is signed, not after a regulator asks.

How Fini is built for this evaluation

Every criterion covered in this guide has a concrete answer at Fini.

Resolution Rate 90% at 99% accuracy across voice, chat, and email, with 3M+ monthly resolutions across fintech and healthcare already in production.

The Zero-Pay Guarantee (Enterprise) lets you test on 1,000 real tickets before any financial commitment: 90% resolution in 90 days, or you pay $0.

Knowledge Atlas handles self-maintenance automatically, running a nightly pipeline that detects gaps, flags conflicting articles, and drafts fixes for human review. Knowledge maintenance drops from roughly 20 hours per week to about 2.

The three-stage rollout maps directly to the pilot structure this guide recommends: Day 1, the Knowledge Agent is live. Day 14, agentic workflows connect to billing, CRM, and backend systems. Day 30, full autonomy with self-learning active.

Pricing is per resolved ticket, $0.49 on the Enterprise plan, with no per-seat fees and free escalations. Compliance coverage: SOC 2 Type II · PCI DSS Level 1 · ISO 27001 · GDPR · HIPAA-compliant · BAA-eligible · CCPA, with every agent decision logged and exportable for audit.

Final thoughts on choosing the right AI customer support vendor

The buying committee, the scorecard, and the pilot structure covered here exist because a compressed timeline and an impressive demo are the two fastest ways to pick the wrong vendor. Resolution rate defined correctly, compliance reviewed upfront, and agentic capability tested on real tickets will tell you more than any scripted walkthrough. Your ops team will thank you for getting those answers before the contract is signed. When you're ready to put a vendor through this process on live traffic, book a 30-minute intro.

FAQ

What should I ask an AI support vendor to verify their resolution rate is real and not inflated?

Ask three questions: what is your re-contact rate within 48 hours, what was the ticket mix behind that number, and is this a cohort-wide figure or a selected subset? A vendor reporting 85% resolution from FAQ-only traffic will not hit that number on billing disputes or account restrictions. Get the methodology in writing, and confirm the contract defines resolution as the customer's issue closed end to end with no human handover.

AI customer support per-resolution vs per-seat pricing: what does total cost actually look like over 12 months?

Per-seat pricing bills regardless of whether the AI resolves anything. Per-conversation pricing runs the meter on every contact, including ones that escalate. Only per-resolution pricing ties what you pay to actual outcomes. To model 12-month cost accurately, you need your current monthly ticket volume, your expected growth rate, and the vendor's definition of a billable event. Confirm whether overages bill at a penalty rate or the same flat rate, and whether unused allowance rolls forward or expires.

How does Fini's Knowledge Atlas handle outdated content and conflicting articles compared to a manual QA process?

Knowledge Atlas runs nightly, detects gaps from escalated tickets, flags conflicts, and queues drafts for review. Manual maintenance averages around 20 hours a week; Atlas brings that down to about 2.

How do I run an AI support pilot that actually predicts production performance?

Send real production tickets including billing disputes, account-state questions, and regulatory edge cases. Define resolution rate, re-contact rate, and accuracy against known ground truth in writing before the pilot starts. Agree on minimum ticket volume and duration upfront, and include tickets the AI should escalate so you can score graceful failure. If a vendor asks to filter the ticket mix before the pilot begins, that request is your finding.

What AI customer support vendor evaluation criteria matter most for fintech and healthcare buyers?

Compliance review belongs at the start of evaluation, not after a demo has won you over. At minimum, require SOC 2 Type II, ISO 27001, PCI DSS for fintech, and both HIPAA-compliant and BAA-eligible for healthcare. Beyond certifications, ask whether the agent's decisions are logged and exportable for audit, and whether the vendor can separate their internal data security posture from whether the agent itself acts in compliance with the regulations governing your business. These are two different questions with two different answers.