A Model Upgrade Is a Breaking Change
A vendor swaps the model behind your API key, on their calendar, and your agent starts answering differently. Nothing in your code changed.
The short version
5 things that decide this
- 01AI providers retire and replace models on their own schedule, not yours, and an alias like "latest" moves you onto the new one automatically.
- 02Behavior can shift even between adjacent versions of the same model family, not only across a full generation change.
- 03A model is a dependency. Treat it like one: pin the exact version, test before upgrading, and roll forward on purpose.
- 04Without a fixed eval suite, you cannot tell whether an upgrade helped, hurt, or did nothing. You can only guess from a few outputs you happened to read.
- 05Write the model version into your logs. When behavior changes, that line is how you find out why.
Nobody changed the code, but the agent changed anyway
A support team notices their AI agent has started giving longer answers than usual. Nobody touched the prompt, and no deploy went out that week. The API endpoint is the same string it has always been.
What changed is on the other end of that endpoint. The provider retired the model version the team had been calling and moved the alias forward to a newer one. The API contract held, but the behavior underneath it did not. Most teams do not plan for this. A model upgrade looks like a vendor doing you a favor, not a change you need to test.
A model is a dependency, and dependencies break things
Software teams already have a name for code you did not write but your product depends on: a dependency. Nobody upgrades a payments library on a Friday afternoon with no tests run. Everyone accepts that a version bump can change behavior. Model providers get treated differently, and the reasoning does not hold up. The prompt you tuned was tuned against one specific model's habits, not against the idea of a model in general.
The shift is often small enough to hide. A model that used to return clean JSON starts adding a line of commentary before it. A parser downstream expected the first character to be a brace, and now it throws on every call. A classifier that reliably picked the right category nine times out of ten drifts to eight. None of this needs a full generation jump. Two adjacent versions of the same model family can differ enough to move a specific task's accuracy, even when the average benchmark score goes up.
Providers give you one real lever here, and most teams do not use it. Call a specific, dated model version instead of a rolling alias. "Use whatever is newest" sounds convenient until newest arrives during your busiest week with no warning and no rollback plan.
- 01A rolling alias moves your agent onto a new model with no deploy, no pull request, and no review.
- 02Output shape can change.
- 03Two versions in the same family can diverge on your specific task even when general benchmarks improve, and a classifier's accuracy on the category you actually care about can drop while the headline score climbs.
Upgrade behind a gate, the way you would any other dependency
Pin the exact model version in one place in the codebase. Scatter it across a dozen call sites and half get missed on the next update. Log which version answered every request. A pattern of complaints then traces back to a specific rollout, not a vague sense that "it got worse recently."
Build a fixed set of real cases with agreed correct answers. Score the candidate model against it before it goes near production. Read the deprecation notice as the start of a short project, not a line to skim. When the score holds, promote the new version deliberately. When it does not, you found that out on your own schedule instead of a customer's.
Questions this raises
01Why did my AI agent's behavior change without a code deploy?
The most common cause is a model version change on the provider's side. A rolling alias, or a retired version replaced by a newer default, swaps the model answering your API calls with nothing in your repository changing. Check your logs for which model version handled the requests before and after the behavior shifted.
02How do I pin an LLM model version to stop it changing under me?
Call a specific, dated model identifier in your API requests instead of a rolling alias like "latest" or the unversioned model name. That version keeps answering the same way until the provider retires it outright. You choose when to move, rather than the provider choosing for you.
03Do I need to test an LLM upgrade the way I test a code change?
Yes, and the test set has to be the same one every time. Run a fixed suite of real cases with known correct answers against the new model version before switching. Compare the score to the version currently live. A model can score higher on general benchmarks and still perform worse on your specific task.
