Retries & Fallbacks

Documentation Index

Fetch the complete documentation index at: https://docs.getbifrost.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Overview

• Retries - When a provider returns a transient server error (network issue, 5xx) or a per-key failure (429 rate-limit, 401/403 auth, 402 billing), Bifrost automatically retries the same request against the same provider. Transient-server retries reuse the same key with exponential backoff; per-key failures rotate to a different API key from your pool. Backoff is skipped only when rotating away from a permanent per-key failure (401/402/403) where waiting offers nothing — for 429 rotations a backoff is still applied to let account-level quota windows slide.
• Fallbacks - When the primary provider fails after exhausting all retries, Bifrost moves on to the next provider in your fallback chain. Each fallback provider gets its own full retry budget.

​

Retries

​

How retries work

1. Classifies the failure as either a per-key failure (the credential / account is the problem — status 401/402/403/429) or a transient server failure (the upstream is the problem — 5xx / network / DNS).
2. On per-key failures, rotates to a different API key from the pool (if multiple keys are configured). Two sub-cases:
   • Permanent per-key failure (401/402/403): mark the key dead for the remainder of the request and rotate immediately — no backoff, since waiting can’t revive a bad credential.
   • Transient per-key failure (429 rate-limit): mark the key as used-this-cycle and rotate, but still apply backoff — providers often enforce account-level quotas shared across keys, so the new key may not have fresh capacity until the window slides.
3. On transient server failures (5xx, DNS, connection refused): reuse the same key and wait using exponential backoff with jitter before the next attempt.
4. Continues until the request succeeds, max_retries is exhausted, or every key is permanently dead (in which case Bifrost returns 502 upstream_credentials_exhausted rather than the raw 4xx, to make it clear the caller’s Bifrost API key is fine — the configured provider credentials are not).

​

Backoff formula

backoff = min(retry_backoff_initial × 2^attempt, retry_backoff_max) × jitter(0.8–1.2)

​

What triggers a retry

​

Configuring retries

• Web UI
• API
• Go SDK
• config.json

Navigate to Providers, select a provider, and open the Network Config section.Set:

• Max Retries - number of additional attempts after the first failure (e.g. 3)
• Retry Backoff Initial - starting backoff in milliseconds (e.g. 500)
• Retry Backoff Max - maximum backoff cap in milliseconds (e.g. 5000)

curl --location 'http://localhost:8080/api/providers' \
--header 'Content-Type: application/json' \
--data '{
    "provider": "openai",
    "keys": [
        {
            "name": "openai-key-1",
            "value": "env.OPENAI_API_KEY",
            "models": ["*"],
            "weight": 1.0
        }
    ],
    "network_config": {
        "max_retries": 3,
        "retry_backoff_initial": 500,
        "retry_backoff_max": 5000
    }
}'

func (a *MyAccount) GetConfigForProvider(provider schemas.ModelProvider) (*schemas.ProviderConfig, error) {
    switch provider {
    case schemas.OpenAI:
        return &schemas.ProviderConfig{
            NetworkConfig: schemas.NetworkConfig{
                MaxRetries:          3,
                RetryBackoffInitial: 500 * time.Millisecond,
                RetryBackoffMax:     5 * time.Second,
            },
            ConcurrencyAndBufferSize: schemas.DefaultConcurrencyAndBufferSize,
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

{
  "providers": {
    "openai": {
      "keys": [
        { "name": "openai-key-1", "value": "env.OPENAI_KEY_1", "models": ["*"], "weight": 1.0 },
        { "name": "openai-key-2", "value": "env.OPENAI_KEY_2", "models": ["*"], "weight": 1.0 },
        { "name": "openai-key-3", "value": "env.OPENAI_KEY_3", "models": ["*"], "weight": 1.0 }
      ],
      "network_config": {
        "max_retries": 3,
        "retry_backoff_initial": 500,
        "retry_backoff_max": 5000
      }
    }
  }
}

Field	Type	Default	Description
max_retries	integer	0	Number of additional attempts after the first failure
retry_backoff_initial	integer (ms)	500	Starting backoff duration in milliseconds
retry_backoff_max	integer (ms)	5000	Maximum backoff cap in milliseconds

​

Key rotation on per-key failures

• 429 Too Many Requests — this key is rate-limited; another may have spare quota.
• 401 Unauthorized / 403 Forbidden — bad / revoked key, or key lacks permission.
• 402 Payment Required — billing issue on this key’s account.

{
  "providers": {
    "openai": {
      "keys": [
        { "name": "openai-key-1", "value": "env.OPENAI_KEY_1", "models": ["*"], "weight": 1.0 },
        { "name": "openai-key-2", "value": "env.OPENAI_KEY_2", "models": ["*"], "weight": 1.0 },
        { "name": "openai-key-3", "value": "env.OPENAI_KEY_3", "models": ["*"], "weight": 1.0 }
      ],
      "network_config": {
        "max_retries": 5
      }
    }
  }
}

​

Fallbacks

​

How fallbacks work

1. Primary attempt: Tries your configured provider with its full retry budget
2. Fallback decision: If the primary fails (and the error is retryable at the provider level), Bifrost moves to the first fallback
3. Sequential fallbacks: Each fallback provider also gets its own full retry budget
4. First success wins: Returns the response from the first provider that succeeds
5. All fail: Returns the original error from the primary provider. Exception: if a plugin on a fallback provider sets AllowFallbacks = false on the error (e.g. a security or compliance plugin that should halt the chain regardless of remaining fallbacks), Bifrost stops immediately and returns that fallback’s error rather than continuing to the next provider or returning the primary error.

​

Implementation

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Explain quantum computing in simple terms"
      }
    ],
    "fallbacks": [
      "anthropic/claude-3-5-sonnet-20241022",
      "bedrock/anthropic.claude-3-sonnet-20240229-v1:0"
    ],
    "max_tokens": 1000,
    "temperature": 0.7
  }'

{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Quantum computing is like having a super-powered calculator..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 150,
    "total_tokens": 162
  },
  "extra_fields": {
    "provider": "anthropic",
    "latency": 1.2
  }
}

package main

import (
    "context"
    "fmt"
    "github.com/maximhq/bifrost"
    "github.com/maximhq/bifrost/core/schemas"
)

func chatWithFallbacks(client *bifrost.Bifrost) {
    ctx := context.Background()

    response, err := client.ChatCompletionRequest(
        schemas.NewBifrostContext(ctx, schemas.NoDeadline),
        &schemas.BifrostChatRequest{
            Provider: schemas.OpenAI,
            Model:    "gpt-4o-mini",
            Input: []schemas.ChatMessage{
                {
                    Role: schemas.ChatMessageRoleUser,
                    Content: &schemas.ChatMessageContent{
                        ContentStr: bifrost.Ptr("Explain quantum computing in simple terms"),
                    },
                },
            },
            // Fallback chain: OpenAI → Anthropic → Bedrock
            Fallbacks: []schemas.Fallback{
                {Provider: schemas.Anthropic, Model: "claude-3-5-sonnet-20241022"},
                {Provider: schemas.Bedrock, Model: "anthropic.claude-3-sonnet-20240229-v1:0"},
            },
            Params: &schemas.ChatParameters{
                MaxCompletionTokens: bifrost.Ptr(1000),
                Temperature:         bifrost.Ptr(0.7),
            },
        },
    )

    if err != nil {
        fmt.Printf("All providers failed: %v\n", err)
        return
    }

    fmt.Printf("Response from %s: %s\n",
        response.ExtraFields.Provider,
        *response.Choices[0].BifrostNonStreamResponseChoice.Message.Content.ContentStr)
}

​

How retries and fallbacks work together

​

Real-world scenarios

​

Plugin execution

• Semantic cache checks run again (the fallback provider may have a cached response)
• Governance rules apply to the new provider
• Logging captures the fallback attempt separately
• All configured plugins execute fresh for each provider in the chain

​

Next steps

• Keys Management - Configure multiple API keys per provider to enable key rotation on retries
• Governance - Use virtual keys and routing rules to control which providers are used
• Observability - Track retry counts and fallback usage in your logs