Fallbacks

​

Automatic Provider Failover

​

How Fallbacks Work

1. Primary Attempt: Tries your main provider/model first
2. Automatic Detection: If the primary fails (network error, rate limit, model unavailable), Bifrost detects the failure
3. Sequential Fallbacks: Tries each fallback provider in order until one succeeds
4. Success Response: Returns the response from the first successful provider
5. Complete Failure: If all providers fail, returns the original error from the primary provider

​

Implementation Examples

# Chat completion with multiple fallbacks
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms"
      }
    ],
    "fallbacks": [
      "anthropic/claude-3-5-sonnet-20241022",
      "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
    ],
    "max_tokens": 1000,
    "temperature": 0.7
  }'

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant", 
        "content": "Quantum computing is like having a super-powered calculator..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 150,
    "total_tokens": 162
  },
  "extra_fields": {
    "provider": "anthropic",
    "latency": 1.2
  }
}

package main

import (
    "context"
    "fmt"
    "github.com/maximhq/bifrost"
    "github.com/maximhq/bifrost/core/schemas"
)

func chatWithFallbacks(client *bifrost.Bifrost) {
    ctx := context.Background()
    
    // Chat request with multiple fallbacks
    response, err := client.ChatCompletionRequest(ctx, &schemas.BifrostChatRequest{
        Provider: schemas.OpenAI,
        Model:    "gpt-4o-mini",
        Input: []schemas.ChatMessage{
            {
                Role: schemas.ChatMessageRoleUser,
                Content: schemas.ChatMessageContent{
                    ContentStr: bifrost.Ptr("Explain quantum computing in simple terms"),
                },
            },
        },
        // Fallback chain: OpenAI → Anthropic → Bedrock
        Fallbacks: []schemas.Fallback{
            {
                Provider: schemas.Anthropic,
                Model:    "claude-3-5-sonnet-20241022",
            },
            {
                Provider: schemas.Bedrock,
                Model:    "anthropic.claude-3-sonnet-20240229-v1:0",
            },
        },
        Params: &schemas.ChatParameters{
            MaxCompletionTokens:   bifrost.Ptr(1000),
            Temperature: bifrost.Ptr(0.7),
        },
    })
    
    if err != nil {
        fmt.Printf("All providers failed: %v\n", err)
        return
    }
    
    // Success! Response came from whichever provider worked
    fmt.Printf("Response from %s: %s\n", 
        response.ExtraFields.Provider, 
        *response.Choices[0].BifrostNonStreamResponseChoice.Message.Content.ContentStr)
}

​

Real-World Scenarios

• Primary: OpenAI hits rate limit → Fallback: Anthropic succeeds
• Your application continues without interruption

• Primary: Specific model unavailable → Fallback: Different provider with similar model
• Seamless transition to equivalent capability

• Primary: Provider experiencing downtime → Fallback: Alternative provider
• Business continuity maintained

• Primary: Premium model for quality → Fallback: Cost-effective alternative if budget exceeded
• Governance rules can trigger fallbacks based on usage

​

Fallback Behavior Details

• Network connectivity issues
• Provider API errors (500, 502, 503, 504)
• Rate limiting (429 errors)
• Model unavailability
• Request timeouts
• Authentication failures

• Request validation errors (malformed requests)
• Plugin-enforced blocks (governance violations)
• Certain provider-specific errors marked as non-retryable

• Semantic cache checks run again (different provider might have cached responses)
• Governance rules apply to the new provider
• Logging captures the fallback attempt
• All configured plugins execute fresh for the fallback provider

• A custom plugin might prevent fallbacks for certain types of errors
• Security plugins might disable fallbacks for compliance reasons