What is a canary deployment?
A bad model update rarely throws an error. It just starts giving worse answers, and a rollout that only watches status codes will not catch that.
Canary deployment
canary release
Canary deployment is a release method that sends a small share of live traffic to a new version. Most traffic stays on the old one. If the canary group's metrics hold up, the rollout widens in steps until everyone is on the new version.
The name comes from coal miners carrying a caged canary underground. A bird is more sensitive to gas than a person, so it showed a problem before anyone got hurt. A canary release does the same job with a small slice of users instead of the whole audience.
Say a team ships version 2 of a checkout service. Instead of switching every user over at once, a router sends 5% of requests to version 2 and keeps the rest on version 1. Someone watches error rates, latency and business metrics on that 5%. If nothing breaks, the share moves to 25%, then 50%, then all of it.
The same idea applies to a model swap. Route a slice of requests to the new model, keep the rest on the model already in production, and compare. The routing looks identical to a code canary. What you have to watch does not.
A model canary needs a different kind of alarm
A code canary fails loudly. A broken deploy throws 500s, and the dashboard everyone already watches lights up within minutes. That is why canary releases for ordinary software got popular: the signal is cheap to collect and hard to miss.
A model swap rarely throws anything. A new model version, a changed prompt or a swapped provider will still return 200 OK on every request. The response is well-formed. It might just be wrong, off-topic or worse at the one task your product depends on. Status codes cannot see that.
In the AI systems we run, the canary metric that matters most is answer quality. Does the new version's output still pass the evals the old one passed?
We have seen teams ship a canary rollout that tracked only errors and latency. It came back clean at 100% traffic. A week later, support started hearing complaints. The assistant had gotten worse at one whole category of question, and nothing on the dashboard had flagged it.
That gap is why a model canary needs its own scorecard before it ships. Run a fixed set of test cases with known-good answers against the canary slice, and compare the score to the baseline. Errors and latency still belong on the dashboard. They are just no longer the whole story.
- Route 5%New version gets a small, real slice of traffic.
- Check errors and latencyThe signal a code canary was built to catch.
- Check answer qualityEval set run against the canary slice for a model swap.
- Widen to 25%, 50%Only after both checks hold at the smaller share.
- Full rollout or rollbackRoute everyone back to the old version if either check fails.
The routing step is identical for code and models. The check step is where a model canary needs more than an error rate.
Common questions
01What is the difference between canary and blue-green deployment?
A canary release sends a small percentage of traffic to the new version and widens gradually, watching metrics at each step. A blue-green deployment keeps two full environments and switches all traffic at once with a single router flip. Canary catches problems earlier, on fewer users; blue-green rolls back faster, because there is no partial state to unwind.
02How big should a canary slice be for a model rollout?
Enough to get a statistically meaningful read on the eval set without exposing too many users to a regression, often starting near 1% to 5%. The right number depends on your traffic volume. Low-traffic products need a longer soak time at a small percentage rather than a bigger slice, or the sample is too small to trust.
03Does a canary deployment need automated rollback?
It should. A canary that requires a human to notice a metric and manually revert loses most of its speed advantage. Set thresholds on both error rate and eval score, and automate the revert when either one is breached during the canary window.
Related terms
- Blue-green deployment →The other release strategy canary gets compared to: two full environments and one switch, instead of a gradual widening.
- Observability →What a canary rollout is actually reading when it decides whether to widen or roll back.
- MLOps consulting →Where we build the eval gates and rollout pipeline a model canary depends on.
