What is model fallback?
A single provider going down takes your product down with it, unless something else answers in its place.
Model fallback
LLM failover
Model fallback routes a request to a second AI model when the primary one is unavailable, rate-limited, or deprecated by its provider. The fallback can come from a different vendor entirely, so it does not share the primary's outages.
A product that calls one model has one point of failure. The provider can have an outage, hit a rate limit on your account, or retire the model version you built against. Every request that depends on it fails at once.
A fallback sits behind the primary call. When the primary errors, times out, or returns a rate-limit response, the request goes to a second model instead. Some teams fall back within one provider's model family. Others route to a different vendor, so one company's outage can't take the product down.
The fallback model is not a copy of the first one
Swapping API keys is the easy part. The prompt, the output format, and the tone were all tuned against the primary model. A different model, even from the same vendor, can follow that same prompt worse.
A prompt engineered to keep one model from over-explaining can make a second model clip its answers short instead. A JSON schema one model follows reliably might get wrapped in prose by another. None of that shows up until the fallback is the one actually answering, which is usually during the exact outage you built it to survive.
Treat the fallback path as a second product, not a spare key. It needs its own evaluation run against the same test cases as the primary, checked before it ever serves a real request, not after.
- 01Rate limits and outages call for a fallback in seconds; a deprecated model version calls for one on a migration schedule, planned in advance.
- 02Cost and latency differ by model too, so a fallback that fires often changes your bill and your response time along with the answers.
- 03Log which model actually answered each request. Without that field, a support ticket about a wrong answer is unsolvable after the fact.
- RequestSent to the primary model.
- CheckError, timeout, or 429 comes back.
- RouteRetry the primary, or switch.
- Fallback modelA different vendor answers instead.
- LogRecord which model served this request.
The switch is the easy half. The eval pass proving the fallback's answers are still acceptable is the half that gets skipped.
Common questions
01Does model fallback mean using a cheaper model to save money?
No, that's a routing decision, not a fallback. A fallback exists for reliability. It only serves traffic when the primary model is unavailable. Some teams combine the two, using a cheaper model as both the default and the failover target, but the two goals should be evaluated separately.
02Should the fallback come from the same provider or a different one?
A different provider protects against a full outage at the vendor level, which a same-provider fallback cannot do. A same-provider fallback is easier to keep consistent in tone and format, since it shares more training lineage with the primary. Many production systems use both: a same-provider fallback first, a different vendor as the last resort.
03How often should a fallback model be re-evaluated?
Any time the primary's prompt changes, and on a fixed schedule regardless. Providers sometimes update a model without changing its version string. A fallback checked six months ago against a prompt that has since changed twice is not verified for what it runs today.
04Is model fallback the same as a circuit breaker?
They're related, but different. A circuit breaker stops sending requests to a failing dependency once it has failed enough times, so retries don't pile onto an outage. Model fallback is what the circuit breaker routes to once it opens: a second model ready to take the traffic.
Related terms
- Model deprecation risk →The slower version of the same problem: a model version retired on the vendor's schedule, not yours.
- OpenAI vs Anthropic →What actually differs between the two vendors a cross-provider fallback usually picks between.
- Software maintenance →Where fallback paths get tested on a schedule instead of left untouched until the outage that needs them.
