Skip to main content

​
Automatic Provider Failover

Fallbacks provide automatic failover when your primary AI provider experiences issues. Whether it’s rate limiting, outages, or model unavailability, Bifrost automatically tries backup providers in the order you specify until one succeeds. When a fallback is triggered, Bifrost treats it as a completely new request - all configured plugins (caching, governance, logging, etc.) run again for the fallback provider, ensuring consistent behavior across all providers.

​
How Fallbacks Work

When you configure fallbacks, Bifrost follows this process:
  1. Primary Attempt: Tries your main provider/model first
  2. Automatic Detection: If the primary fails (network error, rate limit, model unavailable), Bifrost detects the failure
  3. Sequential Fallbacks: Tries each fallback provider in order until one succeeds
  4. Success Response: Returns the response from the first successful provider
  5. Complete Failure: If all providers fail, returns the original error from the primary provider
Each fallback attempt is treated as a fresh request, so all your configured plugins (semantic caching, governance rules, monitoring) apply to whichever provider ultimately handles the request.

​
Implementation Examples

  • Gateway
  • Go SDK
# Chat completion with multiple fallbacks
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms"
      }
    ],
    "fallbacks": [
      "anthropic/claude-3-5-sonnet-20241022",
      "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
    ],
    "max_tokens": 1000,
    "temperature": 0.7
  }'
Response (from whichever provider succeeded):
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant", 
        "content": "Quantum computing is like having a super-powered calculator..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 150,
    "total_tokens": 162
  },
  "extra_fields": {
    "provider": "anthropic",
    "latency": 1.2
  }
}

​
Real-World Scenarios

Scenario 1: Rate Limiting
  • Primary: OpenAI hits rate limit → Fallback: Anthropic succeeds
  • Your application continues without interruption
Scenario 2: Model Unavailability
  • Primary: Specific model unavailable → Fallback: Different provider with similar model
  • Seamless transition to equivalent capability
Scenario 3: Provider Outage
  • Primary: Provider experiencing downtime → Fallback: Alternative provider
  • Business continuity maintained
Scenario 4: Cost Optimization
  • Primary: Premium model for quality → Fallback: Cost-effective alternative if budget exceeded
  • Governance rules can trigger fallbacks based on usage

​
Fallback Behavior Details

What Triggers Fallbacks:
  • Network connectivity issues
  • Provider API errors (500, 502, 503, 504)
  • Rate limiting (429 errors)
  • Model unavailability
  • Request timeouts
  • Authentication failures
What Preserves Original Error:
  • Request validation errors (malformed requests)
  • Plugin-enforced blocks (governance violations)
  • Certain provider-specific errors marked as non-retryable
Plugin Execution: When a fallback is triggered, the fallback request is treated as completely new:
  • Semantic cache checks run again (different provider might have cached responses)
  • Governance rules apply to the new provider
  • Logging captures the fallback attempt
  • All configured plugins execute fresh for the fallback provider
Plugin Fallback Control: Plugins can control whether fallbacks should be triggered based on their specific logic. For example:
  • A custom plugin might prevent fallbacks for certain types of errors
  • Security plugins might disable fallbacks for compliance reasons
When a plugin determines that fallbacks should not be attempted, it can prevent the fallback mechanism entirely, ensuring the original error is returned immediately. This ensures consistent behavior regardless of which provider ultimately handles your request, while giving plugins full control over the fallback decision process. And you can always know which provider handled your request via extra_fields.