What are regression evals?
An engineer rewrites a prompt to fix a customer complaint. The fix works. Three other intents silently start failing, and nobody notices until support tickets climb next week.
Regression evals
Eval regression testing
Regression evals are a fixed set of scored test cases, run automatically every time an AI system's prompt, model or retrieval setup changes. They catch cases that used to pass and now fail. It works the same way a code test suite catches a function broken by an unrelated edit.
The set is not the change itself. It is every case the system has ever been graded on. Each one runs again, whether or not the change was meant to touch it.
A code test suite works the same way. You run the whole suite, because changing one function can break a caller three files away. A prompt behaves the same. One instruction can shift how the model reads every other instruction around it.
A fix that helps one case can break five others
Without regression evals, this is the default outcome, not the exception. A prompt edit that fixes one complaint can change tone, formatting or accuracy on cases nobody was watching. A large language model does not read an instruction in isolation. It reads the whole prompt every time.
The same risk shows up when a model vendor ships a new version, or when a retrieval step starts pulling different documents after a reindex. None of those are prompt edits, but each can move the score on cases that were passing yesterday.
Regression evals turn that risk into a number you see before a release, not a pattern support notices weeks later. The suite runs, the score is compared to the last known-good run, and a drop blocks the release instead of shipping quietly.
- 01Run the full suite on every change, not only the cases the change targeted.
- 02Compare against the last known-good score, not against zero.
- 03Treat a model version bump and a reindex the same as a prompt edit: run the suite before either ships.
- ChangePrompt, model or retrieval edit.
- RunThe whole suite, not the target case.
- CompareAgainst the last known-good score.
- DiffWhich cases moved, and which way.
- GateA drop blocks the release.
The step most teams skip is comparing against a prior score. Without it, a drop just looks like today's result.
Common questions
01How is a regression eval different from an eval-driven development suite?
Eval-driven development is the practice of writing scored test cases before the prompt. Regression evals are what you run afterward, on every later change, using that same set. One is when you build the tests. The other is when you rerun them.
02Do you need regression evals if the model itself does not change?
Yes, because the prompt and the retrieval step change more often than the model does. A new document in the knowledge base or a reworded instruction can shift behaviour on cases that had nothing to do with the change.
03What counts as a regression, if the score is a rate rather than pass or fail?
Set the threshold before you see the new score, not after. A drop below that threshold on the full suite is a regression, even if the specific case you meant to fix improved. Agreeing the number in advance is what stops it becoming whatever you just achieved.
04How often should the suite run?
On every prompt, model or retrieval change, before it reaches production. Some teams also run it on a schedule against live traffic samples, since a vendor can change a hosted model's behaviour without warning.
