LLM Fallback Routing: Survive Model Failures Without Overspending

The chain, and why order matters

Primary: whatever your cost and quality routing chose, say claude-sonnet-4-6.
Same provider sibling: claude-haiku-4-5. No prompt reshaping, no second SDK, and partial outages often leave siblings alive. Slightly worse output beats no output for most features.
Cross provider: for example gpt-5.4-mini or gemini-3.6-flash. Real insurance against a full provider outage, but only if you tested the prompt there before the incident. An untested cross provider fallback is a bug you deploy during an outage.

Keep the chain to two or three entries. Each hop adds worst case latency, and a chain long enough to always answer is also long enough to always answer badly.

What triggers a fallback, and what must not

Fall back on: overloaded responses, 500 class errors, 529s, and timeouts. These say the provider cannot serve you right now.
Do not fall back on: auth errors, malformed requests, context length errors, and content refusals. These say your request is the problem, and the next model will refuse it too, at your expense. Classify first, route second.
Between hops: one short backoff with jitter, not a tight loop. Tight retry loops are how a provider incident becomes your rate limit incident on recovery.

A hand rolled chain is genuinely this small:

const CHAIN = ['claude-sonnet-4-6', 'claude-haiku-4-5', 'gpt-5.4-mini'];

for (const model of CHAIN) {
  try {
    return await callModel(model, prompt);   // your provider dispatch
  } catch (err) {
    if (!isProviderFailure(err)) throw err;  // your bug: do not chain it
    await sleep(300 + Math.random() * 400);
  }
}
throw new Error('all models unavailable');

The cost dimension nobody instruments

Fallback traffic has two cost quirks. Failed primary attempts are not always free, a timeout after partial generation can still bill output tokens. And chains often route upward in price: Haiku at $1 and $5 per million falling back to a $3 and $15 model triples the rate for as long as the chain is active. During an incident nobody is watching spend, which is exactly why the instrumentation has to exist beforehand:

Tag hops distinctly, feature: 'chat' versus feature: 'chat-fallback', so the incident cost is a filter, not a forensic project.
Keep per user caps enforced on the fallback path too, ideally caps that downgrade rather than block, so the chain cannot become an open ended bill. The outage playbook side of this is in what to do when the Claude API is down.
Watch prices on both ends of the chain. They move: one provider cut a model 80 percent in July, per the GPT-5.6 price drop, which can flip which direction your chain routes “upward”. Current rates live at the LLM pricing JSON feed.

FAQ

What is LLM fallback routing?

Automatically retrying a failed LLM call against a backup model or provider, in a pre defined order, so one model being overloaded or down does not take your feature down. It differs from cost routing, which picks the cheapest capable model up front. Fallback answers what if this call fails, cost routing answers which model should get this call. Mature apps do both.

What order should my fallback chain be?

Same provider first, cross provider second. A different model at the same provider needs no prompt reshaping and often survives partial outages, for example Sonnet overloaded while Haiku responds. Cross provider is the real insurance but needs the prompt tested on the second provider ahead of time, since identical prompts behave differently across models.

Which errors should trigger a fallback?

Fall back on overloaded and server errors (429 due to provider load, 500s, 529, timeouts). Do not fall back on your own mistakes: authentication failures, malformed requests, and content policy refusals will fail identically on the next model and just double your latency and cost. Classify the error first, then route.

Does fallback routing increase costs?

Two ways. Failed attempts that returned partial output can still bill tokens, and the fallback model is often pricier than the primary. A chain from a $1 input model to a $5 one quintuples the cost per call while active. Tag fallback calls with their own feature label so you can see exactly what an incident cost you, and cap the fallback path so it cannot run away.

Should I use a gateway or build fallback myself?

A gateway (LiteLLM, Vercel AI Gateway, and similar) gives you fallback chains as configuration and is the fast path if you already route through one. Hand rolling is a thin loop over an ordered model list and keeps the dependency count down. Either way the important work is the same: choosing the chain, testing prompts on the backups, and watching what fallback traffic costs.

Keep reading

Model Routing for LLMs

Send each request to the cheapest model that fits.

Claude API Down? Fallbacks Without Bill Shock

Outage playbook plus the fallback cost multiplier.

Model Downgrade on a Budget

Downgrade the model instead of blocking at a cap.

Route for resilience, then check the receipt

A fallback chain keeps the feature alive. Per user, per feature cost tracking tells you what staying alive cost, which model each call actually landed on, and whether any account ran away during the chaos. Weckr logs all of it in two lines around the client. See it on the live demo, or start from the AI cost and margin guide.

LLM Fallback Routing: Stay Up When a Model Goes Down