LLM Fallback Patterns: Keeping AI Features Up When the Model Provider Goes Down

An LLM fallback strategy has four parts: classify each provider error as retryable or not, retry the retryable ones with capped exponential backoff, fail over to a second model on independent infrastructure when retries run out, and end the chain in a non-AI path that always works, such as a cached answer, a rules-based result or a manual queue. A circuit breaker keeps a failing provider out of the path until it recovers. Most outages then show up to users as a slower or simpler feature rather than a broken one.

A logistics customer called us on a Tuesday afternoon because their AI-powered shipment classifier had stopped responding. Nothing in their code had changed. Their model provider was having a regional incident, requests were timing out after thirty seconds, and the feature that sorted incoming orders had simply gone dark. Support tickets piled up while an engineer refreshed a status page. The fix was not a better model or a bigger budget. It was a set of LLM fallback patterns that the feature should have shipped with on day one, so that when a provider goes down or has a bad afternoon, the feature degrades into a slower experience rather than a dead one.

This is the uncomfortable truth about building on top of large language models: you have taken a hard dependency on an external system that will, at some point, return errors, throttle you, or answer so slowly that the answer is useless. Treating that as an edge case is how you end up explaining an outage you did not cause. Treating it as a design constraint from the start is how you build AI features that stay usable when the thing underneath them wobbles.

Why LLM fallback patterns are not optional

Traditional APIs fail in ways most teams already handle. A payment gateway times out, you retry, you show a friendly error. LLM providers add failure modes that are easy to underestimate. Rate limits arrive in bursts when your traffic spikes, which is exactly when you least want to drop requests. Latency is variable by design, so a call that usually returns in two seconds can take twenty under load. Context windows and token limits reject requests that worked fine yesterday because today's input is longer. And whole regions or model versions can be deprecated or taken offline with little notice.

The blast radius is also different. When an AI feature is core to a workflow, its failure is not cosmetic. The shipment classifier above was not a nice-to-have summary box in a corner of the screen. It sat in the critical path of order intake, so its outage became an operational outage. The more central the feature, the more its reliability has to be engineered deliberately rather than assumed. That engineering work is what separates a demo from a production system, and it is the same discipline we bring to any custom software development project where an external dependency sits on the critical path.

The fallback chain: primary, secondary, degraded

The core pattern is a chain of options ordered from best to acceptable. Your code attempts the first, and on failure moves to the next, until it reaches an option that always works because it does not depend on the provider at all.

A typical chain has three tiers. The primary is your preferred model, chosen for quality. The secondary is a different model, ideally from a different provider, so that a single vendor incident does not take out both links. The final tier is a non-AI fallback: a cached previous answer, a simpler rules-based result, or an honest message that hands control back to the user. The point of the last tier is that it has no external dependency, so the chain can always terminate in something the user can act on.

Provider diversity is what makes the middle tier worth having. If your primary and secondary models are both hosted by the same vendor, a regional outage takes both down together and your chain collapses to the degraded tier immediately. Routing the secondary through an independent provider, even one with a slightly weaker model, means most incidents are invisible to users because the secondary quietly absorbs them. Choosing which providers to pair, and how to keep prompts portable across them, is a tech stack strategy decision worth making before you are in the middle of an incident.

One caution: prompts are not perfectly portable. A prompt tuned for one model's quirks can produce weaker output on another. Keep a provider-neutral version of each prompt and test it against every model in the chain, so the secondary actually produces usable results rather than confidently wrong ones. Validate the structure of what comes back at each tier, because a fallback that returns malformed JSON is not really a fallback.

Classify provider errors before you handle them

Every fallback decision starts with one question: will the same request succeed if you try again, somewhere else, or not at all? The major providers answer it through status codes, but the codes do not mean quite the same thing everywhere. As of September 2026:

Situation	Anthropic	OpenAI	Google Gemini	What to do
Rate limited	429 rate_limit_error with retry-after	429 rate limit codes	429 RESOURCE_EXHAUSTED	Retry after the indicated delay, then fail over
Quota or spend cap reached	429 without retry-after	429 with a quota or spend-limit code	429	Fail over immediately; retrying will not help until the cap resets
Provider overloaded	529 overloaded_error	503	503 UNAVAILABLE	Retry briefly, then fail over
Internal error or timeout	500 api_error, 504 timeout_error	500	500, 504	Retry with backoff, then fail over
Bad request, auth, permissions	400, 401, 403	400, 401, 403	400, 403	Do not retry and do not fail over; alert

Two details from this table cause most of the trouble in practice. First, a 429 is not always temporary. Anthropic and OpenAI both return 429 when an account hits a spend or usage cap, and those requests keep failing until someone raises the cap or the period resets. Treat a 429 without a retry-after header, or with a quota-type error code, as a signal to fail over, not to back off. Second, a bad request that fails on the primary will usually fail on the secondary too. Sending it down the chain only multiplies the error and the cost.

Retry logic that helps instead of hurts

Retrying is the first instinct when a call fails, and done badly it makes outages worse. Immediate retries against a provider that is already overloaded add load to a struggling system and can extend the incident. The discipline that prevents this is exponential backoff with jitter: wait a short interval before the first retry, double it for each subsequent attempt, and add a small random offset so that thousands of clients do not retry in lockstep and create a synchronised thundering herd.

Equally important is knowing what not to retry. A rate-limit response or a transient timeout is worth retrying because the next attempt may well succeed. A malformed-request error or an authentication failure is not, because the same input will fail the same way every time, and retrying just wastes the user's time and your budget. Read the error, classify it as transient or permanent, and only loop on the transient class. Cap the number of attempts and the total time you are willing to spend, because a retry loop that runs longer than the user will wait has failed even if it eventually succeeds.

Check what your SDK already does before you add your own loop. The official Anthropic and OpenAI SDKs retry connection errors, rate limits and server errors automatically, with backoff and respecting retry-after (Anthropic's SDKs default to two retries). Wrap that in an application-level loop with three attempts and a single user request can turn into nine provider calls. Either set the SDK's max-retries option to zero and own the policy in one place, or keep the SDK retries and let your code handle only the failover.

A circuit breaker sits above all of this. When a provider has failed repeatedly in a short window, stop sending it traffic for a cooldown period and route straight to the next tier of the chain. This protects both sides. It stops you from hammering a downed provider, and it stops your users from waiting through a doomed retry sequence on every request. After the cooldown, let a small number of trial requests through to test whether the provider has recovered before you reopen the floodgates.

When the stream breaks halfway

Streaming adds a failure mode that status codes do not cover. With server-sent events the provider returns HTTP 200 as soon as the stream opens, so a failure after the first tokens arrives as an error event or a dropped connection, not as a 5xx. By then the user has already seen part of an answer.

There are three reasonable responses, and the right one depends on the feature. For short answers, discard the partial output and restart the request on the next tier, showing a brief "retrying" state. For long answers, keep what the user has seen, mark it as incomplete and offer a retry button, because restarting silently would replace text the user was reading. For structured output (JSON, tool calls), always discard: a half-finished object is useless, and continuing it on a different model is not safe. What you should not do is stitch the second half of an answer from a different model onto the first, because the two will not agree on tone, facts or format. Our guide to streaming LLM responses in Next.js covers the client side of this.

Build the router yourself or use a gateway?

A fallback chain is a small amount of code for one feature and a real maintenance burden across ten. Three options are common in 2026.

In-application routing keeps the logic in your codebase: a thin client that wraps each provider's SDK, classifies errors as above and walks the chain. It has no extra infrastructure and gives you full control, and it is the right choice while you have one or two AI features.

A self-hosted gateway such as the LiteLLM proxy moves retries, fallbacks and cooldowns into configuration. LiteLLM's router retries a failing model, falls back to the next model in the chain when retries run out, and puts a deployment into cooldown after repeated failures, which is the circuit breaker described above. You run and monitor one more service, and in exchange every feature gets the same policy.

A hosted router such as OpenRouter gives you one API across many providers with failover built in. It is the fastest to adopt, but it adds a third party to your data path, which matters for GDPR and for enterprise customers who ask where their data goes.

One option is easy to overlook: the same model through a different cloud. Claude models are available directly from Anthropic and through Amazon Bedrock and Google Cloud Vertex AI; OpenAI models are available directly and through Microsoft Azure. A secondary tier that runs the same model on different infrastructure avoids the prompt-portability problem entirely, and it can let you keep processing inside an EU region. We compare the architectures in more depth in LLM gateway architecture.

Graceful degradation is a product decision

The most important part of resilience is deciding, before an incident, what "still working" means for each feature when the AI layer is unavailable. This is a product question as much as an engineering one, and it deserves a real answer rather than a generic error page.

Graceful degradation usually takes one of a few shapes. A cached or recent result can stand in when freshness is not critical, so a user sees yesterday's summary rather than nothing. A simpler deterministic path can replace the model entirely, such as keyword rules where an LLM normally classifies, accepting lower accuracy in exchange for guaranteed availability. Or the feature can step aside cleanly and let the user do the task manually, which is far better than a spinner that never resolves. The classifier customer landed on a manual queue: when the model is unavailable, new orders drop into a list a human can sort, and the system tells the operator exactly that. Throughput drops, but the business keeps running.

Whatever shape you choose, be honest in the interface. A short, specific message that the AI assist is temporarily unavailable, paired with a working manual path, preserves trust. Silent failures and infinite spinners destroy it. The teams that handle this well decide the degraded experience deliberately and design for it, the same way they would design any other web application state, rather than discovering it live during an outage.

Make failure observable before it happens

You cannot manage failures you cannot see. Instrument every call to the provider with its outcome, latency, which tier of the chain served the response, and the token counts involved. This turns a vague sense that "the AI feels slow today" into a concrete signal you can alert on. When fallbacks start firing more often than usual, that is your early warning that a provider is degrading, often before their own status page admits it.

Record the provider's request ID with every call (Anthropic returns it in the request-id header, and the other providers have equivalents), because support will ask for it when you report an incident. Set thresholds that trigger alerts on elevated error rates, rising latency, and unusual fallback frequency, and log enough context to reconstruct what happened after the fact. A short runbook that names who responds, how to force traffic onto the secondary provider, and how to verify recovery turns a scramble into a procedure. Building that observability and the resilience patterns underneath it is exactly the kind of work a code quality consulting engagement surfaces, because the gaps are rarely in the happy path and almost always in the error handling that no one tested under real failure.

Where to start

If your AI feature has no fallback today, you do not need to build all of this at once. Start with the single highest-impact step: add a non-AI degraded tier so the feature can never fully die, even if that tier is just a clear message and a manual path. Then classify errors and add retry logic with backoff for the transient ones, since that alone absorbs a large share of real-world blips. Add a second provider once the feature is important enough to justify the cost of keeping prompts portable. Layer in circuit breaking and observability as traffic grows. Each step is independently valuable, and the order keeps you protected against the worst outcome first.

The goal is not a perfect system that never sees a provider fail. Providers will fail, and that is outside your control. The goal is that when they do, your users experience a slower or simpler version of the feature instead of an outage, and your team gets a calm alert instead of a support flood.

If you are building AI features into a product and want a second set of eyes on the resilience design, or you are already living through the kind of afternoon described at the top of this post, we are happy to help. Reach out at hello@wolf-tech.io or read more about how we work at wolf-tech.io.