AI evals

What are AI evals?

AI evals are structured tests that measure how accurately, safely, and consistently an AI system responds, scored against a fixed set of cases before a release ships and repeatedly after it goes live. Each case pairs an input with an expected outcome and an explicit scoring rule.

Evals borrow their shape from software testing, but the thing under test is probabilistic: the same question can produce two different answers on two runs. A single passing example proves very little, so teams score datasets of dozens to thousands of cases and watch the aggregate move between versions.

How AI evals work

An eval harness has four parts and one loop. The dataset holds cases: an input, the context the system is allowed to see, and the behavior expected in response. The runner replays every case against one pinned version of the system, fixing the model, the retrieval index, and the prompts together, since a change in prompt engineering can move a score as far as a model swap does. The scorer applies a rule to each output: exact match for structured fields, similarity for free text, and a rubric graded by an LLM as a judge for anything subjective. The gate converts scores into a release decision, usually a pass threshold per category plus a hard block on any safety failure.

The loop runs in production. Live conversations are sampled, turns carrying a low confidence score are pulled first, and the ones the system handled badly are labeled and added to the dataset. Every incident becomes a permanent regression case.

Types of AI evals

Five categories, run at different points in the lifecycle:

• Golden-set evals: A curated file of known inputs and approved answers, replayed on every change to catch regressions, though it only covers what someone thought to write down.

• Production evals: Sampled live conversations scored after the fact, which reveal the real distribution of questions rather than the imagined one.

• Adversarial evals: Deliberately hostile inputs such as prompt injection, jailbreak attempts, and out-of-scope requests, used to test refusal behavior under pressure.

• Action evals: Checks on what the agent did (refund issued, ticket routed, record updated), scored on the side effect, since a fluent answer attached to a wrong API call still fails.

• Human review evals: Sampled transcripts graded by trained reviewers, the human in the loop layer that calibrates every automated scorer above it.

AI evals vs AI agent testing vs customer service QA vs public benchmarks

Four practices overlap here, and teams routinely buy one expecting another. AI agent testing verifies that one deployed agent handles its known cases and permitted actions correctly. Customer service QA scores completed conversations against a service scorecard covering tone, policy adherence, and resolution. Public benchmarks rank models against shared academic datasets that know nothing about your policies. AI evals are the measurement layer underneath all three: the datasets, the scorers, and the thresholds those practices consume.

If you are choosing a base model, read the public benchmarks. If you are managing service quality, run QA. If you need to know whether this week's change made your agent better or worse, evals are the only one of the four that answers it.

Why AI evals matter for customer experience

When evals are absent, quality becomes anecdotal. The loudest escalation sets the roadmap, a regression that breaks a tenth of refund answers hides behind an aggregate satisfaction score that moved half a point, and nobody can tell whether the fault sits in the model, the retrieved article, or the prompt. Teams roll back changes they cannot diagnose, then freeze the system because every edit starts to feel risky.

Customers experience that freeze as inconsistency. The same policy question gets three different answers across a month of releases, and an agent that handled cancellations well in March quietly stops citing the fee condition in May.

The tradeoff is real. Every case in the dataset is content someone has to maintain, and a golden set built once and left alone keeps passing while the policies it encodes expire, producing confidence at exactly the moment a team stops looking.

How are AI evals measured?

Evals get graded on three properties: pass rate, agreement, and coverage. Pass rate is the share of cases meeting their scoring rule, reported per category so a safety failure never averages away inside a strong accuracy number. Agreement is how often an automated scorer matches a trained human on the same sample, checked across a few hundred labeled cases before that scorer is trusted to run unattended.

Coverage is the property teams underestimate, and academic work makes the gap visible. In the Holistic Evaluation of Language Models project, Liang et al. measured 30 language models across 42 scenarios and raised the density of that grid, the share of model and scenario pairs actually reported, from 17.9% to 96.0%. Most internal suites sit nearer the first number: a handful of intents measured well, the rest untested and assumed fine.

How AI agents change AI evals

An agent's output is no longer a paragraph. It is a sequence: retrieve a policy, call an order API, decide, then write. Scoring only the final text passes a reply that reads beautifully and confirms a refund the agent never issued. Evals for agents therefore score the trajectory, checking which tools were called, with what arguments, in what order, and whether the agent stopped and handed off when it should have.

Multi-step work also compounds error. A chain of five steps, each correct 95% of the time, finishes correctly about 77% of the time, so per-step accuracy that looks acceptable produces a task success rate that is not.

Because retrieval indexes and upstream models shift underneath a running agent, model drift turns evals from a launch gate into a continuous monitor. Teams recovering from failed automations usually begin by rebuilding the dataset out of the transcripts that went wrong.

How to run AI evals in practice

Start with the dataset, since everything else is downstream of it. Pull three months of transcripts, cluster them by contact reason, and write cases for the reasons carrying the most volume and the most risk. Coverage is then measured against that contact-reason list.

Integration surface decides whether the suite survives. The runner needs the same retrieval index, the same tool endpoints, and a sandboxed copy of any system the agent writes to, and it should fire from the deployment pipeline without anyone remembering to trigger it.

Governance is ownership: a named owner per category, a review date on every case, and a record of which version produced which score. Regulated buyers ask how those records are evidenced, and frameworks such as ISO 42001 and SOC 2 Type II are where that question usually lands.

The constraint that bites hardest is scorer drift. When a judge model is upgraded quietly, last month's scores stop being comparable, so pin the judge version and re-baseline on purpose.

AI evals and customer service quality assurance

Customer service quality assurance and AI evals ask the same question from opposite ends of the conversation. QA samples finished contacts and grades them against a service scorecard, which tells you how the last thousand conversations went. Evals replay a fixed set of cases against a specific build, which tells you what the next thousand will look like if you ship it.

The two feed each other. QA reviewers flag the conversations that went wrong, and those transcripts become eval cases, while eval failures tell reviewers which intents deserve a heavier sample. AI agent testing is where that combined dataset gets exercised before a release reaches customers.

What do AI evals mean in plain terms?

Think of AI evals as a driving test you can give the same car after every repair. The route never changes, so when the car passes on Monday and fails on Friday, you know the work done on Wednesday caused it. The test does not prove the car is safe on every road. It proves that something changed, and when.

Without that fixed route, the first person to notice a change is a customer, and the report arrives as a complaint about one conversation. Nobody can say whether it was a one-off or the first of four hundred.

The tradeoff is that somebody has to write the route and keep it current. Building cases is slow, unglamorous work that produces no visible feature, and it is the first thing cut when a launch date moves. Teams that cut it find out later, and from someone else.

Common AI eval mistakes

Tuning on the test set. When the same dataset drives both prompt iteration and release approval, the score stops measuring generalization and starts measuring memorization. Hold out a slice nobody is allowed to edit against.

Scoring words while ignoring actions. A text-similarity scorer will happily pass a polite confirmation of a refund that was never issued, which is why action checks and guardrails in agentic support belong in the same suite as answer grading.

Trusting an ungraded judge. A judge model that has never been compared against human reviewers on a labeled sample is an opinion with a decimal point, and its preference for long fluent answers will quietly reshape the system it grades.

Running evals once. A suite built for launch day and never re-run gives a team the paperwork of testing with none of the protection, because the content, the model, and the policies all moved afterward.