What are AI evals?
Without them you edit one line of a prompt, and six weeks later you learn the system has been reading the wrong column on every payroll file.
AI evals
EvaluationsLLM evals
AI evals are a graded set of test cases run against a model, scoring its output against a known correct answer. They tell a team whether a prompt, model or data change made the system better or worse.
An eval has three parts. A dataset of inputs with the answer you expect. A grader that decides whether the output matches. A score you can compare against last week's run.
Graders vary by task. Pulling a gross payroll figure out of a register has one right answer, so an exact match works. Summarising a claim file does not. There you score against a rubric, or hand it to a second model told what good looks like, and sample the results yourself.
A prompt is a dependency you cannot see
Ordinary software fails loudly. A model fails quietly and stays plausible. Change one instruction to fix a rare document type, and the fix can quietly cost you accuracy on the common one. Nothing throws. The output still looks right.
Evals turn that invisible change into a number that moved. They also answer a question a regulated buyer will ask: how do you know it still works? Pointing at a passing suite with dates on it is a better answer than describing your process.
This matters most in document-heavy insurance work. Reconciling a payroll register against an IRS Form 941 has a checkable right answer, which means accuracy is measurable and drift is catchable.
- 01Run them on every prompt change, model version bump and retrieval tweak.
- 02Keep the failures. A case the system got wrong once is the most valuable test you own.
- 03Score the cases you would be embarrassed to get wrong, not the easy majority.
- CollectReal inputs, not invented ones.
- LabelA human sets the right answer.
- RunModel output, every change.
- GradeExact match, or a rubric.
- GateScore drops, release stops.
- AddEvery new failure joins the set.
The gate is the part teams skip. A suite that reports a score but never blocks a release is a dashboard, not a test.
Common questions
01How are evals different from unit tests?+
A unit test asserts one exact output and fails if a single character differs. An eval scores a distribution, because the same input can produce different wording each run. So evals report a rate across many cases rather than a pass or fail on one, and the gate is a threshold you set.
02How many test cases do you need?+
Fewer than teams expect, if they are the right ones. A few dozen genuinely difficult, correctly labelled cases catch more regressions than a thousand easy ones. Labelling cost is the real constraint, since a human has to decide the right answer for each.
03Can a model grade its own output?+
For open-ended tasks, yes, with checks. A separate model given a clear rubric grades summaries and answers reasonably well, and it is far cheaper than a human. Sample its verdicts by hand regularly, because a grader can drift the same way the system under test can.
04Do evals prove the system is compliant?+
No. Evals prove accuracy and catch drift. Compliance also needs logged inputs and outputs, versioned prompts, recorded human overrides and a defined escalation point. Evals are one of the artefacts an examiner finds convincing, not the whole file.
