Documentation Index

Fetch the complete documentation index at: https://docs.getbifrost.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

Overview

• Multi-model support - Unified interface for Gemini, Anthropic, and third-party models
• OAuth2 authentication - Service account credentials with automatic token refresh
• Project and region management - Automatic endpoint construction from GCP project/region
• Model routing - Automatic provider detection (Gemini vs Anthropic) based on model name
• Request conversion - Conversion to underlying provider format (Gemini or Anthropic)
• Embeddings support - Vector generation with task type and truncation options
• Model discovery - Paginated model listing with deployment information

​

Supported Operations

​

Setup & Configuration

​

1. Service Account JSON (Recommended for Production)

• Web UI
• API
• config.json
• Go SDK

1. Navigate to “Model Providers” → “Configurations” → “Google Vertex”
2. Click “Add Key” (or edit an existing key)
3. Under Authentication Method, select “Service Account (JSON)”
4. Set Project ID: Your Google Cloud project ID
5. Set Project Number (Required only for fine-tuned models): Your GCP project number; leave blank for standard models
6. Set Region: e.g., us-central1
7. Set Auth Credentials: Paste your service account JSON or reference an env var (e.g., env.VERTEX_CREDENTIALS)
8. Configure Aliases: Map model names to fine-tuned model IDs (if using fine-tuned models)
9. Save

# Step 1: Create the provider
curl -X POST http://localhost:8080/api/providers \
  -H "Content-Type: application/json" \
  -d '{"provider": "vertex"}'

# Step 2: Create a key (Service Account JSON)
curl -X POST http://localhost:8080/api/providers/vertex/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "vertex-sa-key",
    "value": "",
    "models": ["*"],
    "weight": 1.0,
    "vertex_key_config": {
      "project_id": "env.VERTEX_PROJECT_ID",
      "region": "us-central1",
      "auth_credentials": "env.VERTEX_CREDENTIALS"
    }
  }'

On v1.4.x, two differences apply: - Pass keys directly in the POST /api/providers body - there is no separate /api/providers/{provider}/keys endpoint. - Use deployments inside vertex_key_config instead of the top-level aliases field for fine-tuned model mappings.

{
  "providers": {
    "vertex": {
      "keys": [
        {
          "name": "vertex-sa-key",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "vertex_key_config": {
            "project_id": "env.VERTEX_PROJECT_ID",
            "region": "us-central1",
            "auth_credentials": "env.VERTEX_CREDENTIALS"
          }
        }
      ]
    }
  }
}

On v1.4.x, use deployments inside vertex_key_config instead of the top-level aliases field for fine-tuned model mappings.

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.Vertex:
        return []schemas.Key{
            {
                Value:  schemas.EnvVar{}, // Leave empty when using service account credentials
                Models: []string{"*"},
                Weight: 1.0,
                VertexKeyConfig: &schemas.VertexKeyConfig{
                    ProjectID:       *schemas.NewEnvVar("env.VERTEX_PROJECT_ID"),
                    Region:          *schemas.NewEnvVar("us-central1"),
                    AuthCredentials: *schemas.NewEnvVar("env.VERTEX_CREDENTIALS"), // full service account JSON
                },
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

​

2. Application Default Credentials

1. GOOGLE_APPLICATION_CREDENTIALS env var (path to a JSON credential file)
2. Application default credential file (~/.config/gcloud/application_default_credentials.json, written by gcloud auth application-default login)
3. GCE/GKE/Cloud Run/App Engine metadata server (attached service account or Workload Identity)

• Web UI
• API
• config.json
• Go SDK

1. Navigate to “Model Providers” → “Configurations” → “Google Vertex”
2. Click “Add Key” (or edit an existing key)
3. Under Authentication Method, select “Service Account (Attached)”
4. Set Project ID: Your Google Cloud project ID
5. Set Project Number (Required only for fine-tuned models): Your GCP project number; leave blank for standard models
6. Set Region: e.g., us-central1
7. Configure Aliases if needed
8. Save

Ensure GOOGLE_APPLICATION_CREDENTIALS is set in your environment, or that Workload Identity / gcloud is configured.

# Step 1: Create the provider
curl -X POST http://localhost:8080/api/providers \
  -H "Content-Type: application/json" \
  -d '{"provider": "vertex"}'

# Step 2: Create a key (Application Default Credentials)
curl -X POST http://localhost:8080/api/providers/vertex/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "vertex-adc-key",
    "value": "",
    "models": ["*"],
    "weight": 1.0,
    "vertex_key_config": {
      "project_id": "env.VERTEX_PROJECT_ID",
      "region": "us-central1",
      "auth_credentials": ""
    }
  }'

On v1.4.x, pass keys directly in the POST /api/providers body - there is no separate /api/providers/{provider}/keys endpoint.

{
  "providers": {
    "vertex": {
      "keys": [
        {
          "name": "vertex-adc-key",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "vertex_key_config": {
            "project_id": "env.VERTEX_PROJECT_ID",
            "region": "us-central1",
            "auth_credentials": ""
          }
        }
      ]
    }
  }
}

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.Vertex:
        return []schemas.Key{
            {
                Value:  schemas.EnvVar{},
                Models: []string{"*"},
                Weight: 1.0,
                VertexKeyConfig: &schemas.VertexKeyConfig{
                    ProjectID: *schemas.NewEnvVar("env.VERTEX_PROJECT_ID"),
                    Region:    *schemas.NewEnvVar("us-central1"),
                    // Leave AuthCredentials empty - uses Application Default Credentials
                },
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

​

3. API Key (Gemini and Fine-Tuned Models Only)

• Web UI
• API
• config.json
• Go SDK

1. Navigate to “Model Providers” → “Configurations” → “Google Vertex”
2. Click “Add Key” (or edit an existing key)
3. Under Authentication Method, select “API Key”
4. Set API Key: Your Vertex AI API key
5. Set Project ID: Your Google Cloud project ID
6. Set Project Number (Required only for fine-tuned models): Your GCP project number; leave blank for standard models
7. Set Region: e.g., us-central1
8. Configure Aliases: Map short names to fine-tuned model IDs (e.g., my-model → 123456789)
9. Save

# Step 1: Create the provider
curl -X POST http://localhost:8080/api/providers \
  -H "Content-Type: application/json" \
  -d '{"provider": "vertex"}'

# Step 2: Create a key (API Key - Gemini + fine-tuned models)
curl -X POST http://localhost:8080/api/providers/vertex/keys \
  -H "Content-Type: application/json" \
  -d '{
    "name": "vertex-api-key",
    "value": "env.VERTEX_API_KEY",
    "models": ["gemini-pro", "gemini-2.0-flash", "my-fine-tuned-model"],
    "weight": 1.0,
    "aliases": {
      "my-fine-tuned-model": "123456789"
    },
    "vertex_key_config": {
      "project_id": "env.VERTEX_PROJECT_ID",
      "project_number": "env.VERTEX_PROJECT_NUMBER",
      "region": "us-central1"
    }
  }'

On v1.4.x, two differences apply:

• Pass keys directly in the POST /api/providers body - there is no separate /api/providers/{provider}/keys endpoint.
• Replace the top-level aliases with "deployments" inside vertex_key_config:

"vertex_key_config": {
  "project_id": "env.VERTEX_PROJECT_ID",
  "region": "us-central1",
  "deployments": {
    "my-fine-tuned-model": "123456789"
  }
}

{
  "providers": {
    "vertex": {
      "keys": [
        {
          "name": "vertex-api-key",
          "value": "env.VERTEX_API_KEY",
          "models": ["gemini-pro", "gemini-2.0-flash", "my-fine-tuned-model"],
          "weight": 1.0,
          "aliases": {
            "my-fine-tuned-model": "123456789"
          },
          "vertex_key_config": {
            "project_id": "env.VERTEX_PROJECT_ID",
            "project_number": "env.VERTEX_PROJECT_NUMBER",
            "region": "us-central1"
          }
        }
      ]
    }
  }
}

On v1.4.x, use deployments inside vertex_key_config instead of the top-level aliases field.

func (a *MyAccount) GetKeysForProvider(ctx *context.Context, provider schemas.ModelProvider) ([]schemas.Key, error) {
    switch provider {
    case schemas.Vertex:
        return []schemas.Key{
            {
                Value:  *schemas.NewEnvVar("env.VERTEX_API_KEY"), // only when using Gemini or fine-tuned models
                Models: []string{"gemini-pro", "gemini-2.0-flash", "my-fine-tuned-model"},
                Weight: 1.0,
                Aliases: schemas.KeyAliases{
                    "my-fine-tuned-model": "123456789",
                },
                VertexKeyConfig: &schemas.VertexKeyConfig{
                    ProjectID:     *schemas.NewEnvVar("env.VERTEX_PROJECT_ID"),
                    ProjectNumber: *schemas.NewEnvVar("env.VERTEX_PROJECT_NUMBER"), // required for fine-tuned models
                    Region:        *schemas.NewEnvVar("us-central1"),
                },
            },
        }, nil
    }
    return nil, fmt.Errorf("provider %s not supported", provider)
}

​

GKE Workload Identity Federation

1. The GCP-side prerequisites below (API enabled, IAM service account, WIF binding)
2. A Bifrost Vertex key using “Service Account (Attached)” auth - see Application Default Credentials for Web UI, API, config.json, and Go SDK setup. For Helm, see Helm - Google Vertex AI.
3. The Kubernetes ServiceAccount annotated for WIF:

kubectl annotate serviceaccount KSA_NAME \
  --namespace NAMESPACE \
  iam.gke.io/gcp-service-account=IAM_SA_NAME@PROJECT_ID.iam.gserviceaccount.com

​

GCP Prerequisites

gcloud services enable aiplatform.googleapis.com --project=PROJECT_ID

gcloud services enable iamcredentials.googleapis.com --project=PROJECT_ID

# Enable Workload Identity on the cluster
gcloud container clusters update CLUSTER_NAME \
  --location=LOCATION \
  --workload-pool=PROJECT_ID.svc.id.goog

# Enable GKE metadata server on each node pool
gcloud container node-pools update NODEPOOL_NAME \
  --cluster=CLUSTER_NAME \
  --location=LOCATION \
  --workload-metadata=GKE_METADATA

gcloud container clusters describe CLUSTER_NAME \
  --location=LOCATION \
  --format="value(workloadIdentityConfig.workloadPool)"
# Expected: PROJECT_ID.svc.id.goog

# Create the service account
gcloud iam service-accounts create IAM_SA_NAME \
  --display-name="Bifrost Vertex AI" \
  --project=PROJECT_ID

# Grant Vertex AI access
gcloud projects add-iam-policy-binding PROJECT_ID \
  --member="serviceAccount:IAM_SA_NAME@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

gcloud iam service-accounts add-iam-policy-binding \
  IAM_SA_NAME@PROJECT_ID.iam.gserviceaccount.com \
  --role="roles/iam.workloadIdentityUser" \
  --member="serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

kubectl annotate serviceaccount KSA_NAME \
  --namespace NAMESPACE \
  iam.gke.io/gcp-service-account=IAM_SA_NAME@PROJECT_ID.iam.gserviceaccount.com

​

Verify

kubectl exec -n NAMESPACE POD_NAME -- \
  wget -qO- --header="Metadata-Flavor: Google" \
  "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token"

​

Troubleshooting

​

Beta Headers

​

1. Chat Completions

​

Request Parameters

​

Core Parameter Mapping

​

Key Configuration

{
  "vertex_key_config": {
    "project_id": "my-gcp-project",
    "region": "us-central1",
    "auth_credentials": "{service-account-json}"
  }
}

• project_id - GCP project ID (required)
• region - GCP region for API endpoints (required)
  • Examples: us-central1, us-west1, eu-west1, global
• auth_credentials - Service account JSON credentials (optional if using default credentials)

​

Authentication Methods

1. Service Account JSON (recommended for production)
   { "auth_credentials": "{full-service-account-json}" }
   
2. Application Default Credentials (for local development)
   • Requires GOOGLE_APPLICATION_CREDENTIALS environment variable
   • Leave auth_credentials empty

​

Gemini Models

​

Parameter Mapping for Gemini

• System prompts: Converted to Gemini’s system message format
• Tool usage: Mapped to Gemini’s function calling format
• Streaming: Uses Gemini’s streaming protocol

​

Anthropic Models (Claude)

​

Parameter Mapping for Anthropic

• Reasoning/Thinking: reasoning parameters converted to thinking structure
• System messages: Extracted and placed in separate system field
• Tool message grouping: Consecutive tool messages merged
• API version: Automatically set to vertex-2023-10-16 for Anthropic models

​

Special Notes for Vertex + Anthropic

• Responses API uses special /v1/messages endpoint
• anthropic_version automatically set to vertex-2023-10-16
• Minimum reasoning budget: 1024 tokens
• Model field removed from request (Vertex uses different identification)

​

Region Selection

​

Streaming

• Gemini models: Standard Gemini streaming with server-sent events
• Anthropic models: Anthropic message streaming format

​

2. Responses API

​

Request Parameters

​

Core Parameter Mapping

​

Gemini Models

​

Anthropic Models (Claude)

• instructions becomes system message
• reasoning mapped to thinking structure

​

Configuration

curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vertex/claude-3-5-sonnet",
    "input": "What is AI?",
    "instructions": "You are a helpful assistant",
    "project_id": "my-gcp-project",
    "region": "us-central1"
  }' \
  -H "X-Goog-Authorization: Bearer {token}"

resp, err := client.ResponsesRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostResponsesRequest{
    Provider: schemas.Vertex,
    Model:    "claude-3-5-sonnet",
    Input:    messages,
    Params: &schemas.ResponsesParameters{
        Instructions: schemas.Ptr("You are a helpful assistant"),
    },
})

​

Special Handling

• Endpoint: /v1/messages (Anthropic format)
• anthropic_version set to vertex-2023-10-16 automatically
• Model and region fields removed from request
• Raw request body passthrough supported

​

3. Embeddings

​

Request Parameters

​

Core Parameters

​

Advanced Parameters

curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-004",
    "input": ["text to embed"],
    "dimensions": 256,
    "task_type": "RETRIEVAL_DOCUMENT",
    "title": "Document title",
    "project_id": "my-gcp-project",
    "region": "us-central1",
    "autoTruncate": true
  }'

resp, err := client.EmbeddingRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostEmbeddingRequest{
    Provider: schemas.Vertex,
    Model:    "text-embedding-004",
    Input: &schemas.EmbeddingInput{
        Texts: []string{"text to embed"},
    },
    Params: &schemas.EmbeddingParameters{
        Dimensions: schemas.Ptr(256),
        ExtraParams: map[string]interface{}{
            "task_type": "RETRIEVAL_DOCUMENT",
            "title": "Document title",
            "autoTruncate": true,
        },
    },
})

​

Embedding Parameters

​

Task Type Effects

• RETRIEVAL_DOCUMENT - Optimized for documents in retrieval systems
• RETRIEVAL_QUERY - Optimized for queries searching documents
• SEMANTIC_SIMILARITY - Optimized for semantic similarity tasks
• CLASSIFICATION - For classification tasks
• CLUSTERING - For clustering tasks

​

Response Conversion

{
  "embeddings": [
    {
      "values": [0.1234, -0.5678, ...],
      "statistics": {
        "token_count": 15,
        "truncated": false
      }
    }
  ]
}

• values - Embedding vector as floats
• statistics.token_count - Input token count
• statistics.truncated - Whether input was truncated due to length

​

4. Image Generation

​

Request Parameters

​

Core Parameter Mapping

​

Model Type Detection

1. Gemini Models: Uses Gemini format (same as Gemini Image Generation)
2. Imagen Models: Uses Imagen format (detected via IsImagenModel())

​

Configuration

curl -X POST http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vertex/imagen-4.0-generate-001",
    "prompt": "A sunset over the mountains",
    "size": "1024x1024",
    "n": 2,
    "project_id": "my-gcp-project",
    "region": "us-central1"
  }' \
  -H "X-Goog-Authorization: Bearer {token}"

resp, err := client.ImageGenerationRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostImageGenerationRequest{
    Provider: schemas.Vertex,
    Model:    "imagen-4.0-generate-001",
    Input: &schemas.ImageGenerationInput{
        Prompt: "A sunset over the mountains",
    },
    Params: &schemas.ImageGenerationParameters{
        Size: schemas.Ptr("1024x1024"),
        N:    schemas.Ptr(2),
    },
})

​

Request Conversion

• Gemini Models: Uses gemini.ToGeminiImageGenerationRequest() - same conversion as standard Gemini (see Gemini Image Generation)
• Imagen Models: Uses gemini.ToImagenImageGenerationRequest() - Imagen-specific format with size/aspect ratio conversion

​

Response Conversion

• Gemini Models: Responses converted using GenerateContentResponse.ToBifrostImageGenerationResponse() - same as standard Gemini
• Imagen Models: Responses converted using GeminiImagenResponse.ToBifrostImageGenerationResponse() - Imagen-specific format

​

Endpoint Selection

• Fine-tuned models: /v1beta1/projects/{projectNumber}/locations/{region}/endpoints/{deployment}:generateContent
• Imagen models: /v1/projects/{projectID}/locations/{region}/publishers/google/models/{model}:predict
• Gemini models: /v1/projects/{projectID}/locations/{region}/publishers/google/models/{model}:generateContent

​

Streaming

​

5. Image Edit

1. Gemini Models: Uses gemini.ToGeminiImageEditRequest() - same conversion as standard Gemini (see Gemini Image Edit)
2. Imagen Models: Uses gemini.ToImagenImageEditRequest() - Imagen-specific format with edit mode mapping and mask configuration (see Gemini Image Edit)

• All request bodies are converted to map[string]interface{} for Vertex API compatibility
• The region field is removed before sending to Vertex API
• For Gemini models, unsupported fields are stripped via stripVertexGeminiUnsupportedFields() (removes id from function_call and function_response)

• Gemini Models: Responses converted using GenerateContentResponse.ToBifrostImageGenerationResponse() - same as standard Gemini
• Imagen Models: Responses converted using GeminiImagenResponse.ToBifrostImageGenerationResponse() - Imagen-specific format

• Gemini models: /v1/projects/{projectID}/locations/{region}/publishers/google/models/{model}:generateContent
• Imagen models: /v1/projects/{projectID}/locations/{region}/publishers/google/models/{model}:predict

​

6. List Models

​

Request Parameters

​

Response Conversion

{
  "models": [
    {
      "name": "projects/{project}/locations/{region}/models/gemini-2.0-flash",
      "display_name": "Gemini 2.0 Flash",
      "description": "Fast multimodal model",
      "version_id": "1",
      "version_aliases": ["latest", "stable"],
      "capabilities": [...],
      "deployed_models": [...]
    }
  ],
  "next_page_token": "..."
}

​

Custom vs Non-Custom Models

​

Three-Pass Model Discovery

1. First Pass - Custom Models from API Response
   • Queries Vertex AI’s List Models API
   • Returns only custom fine-tuned models deployed to your project
   • Custom models are identified by having deployment values that contain only digits
   • Example: "deployment": "1234567890"
2. Second Pass - Non-Custom Models from Aliases
   • Adds standard foundation models from your aliases configuration
   • Non-custom models have alphanumeric deployment values (e.g., gemini-pro, claude-3-5-sonnet)
   • Filters by the key-level models allowlist, if specified
   • Example: "deployment": "gemini-2.0-flash"
3. Third Pass - Allowed Models Not in Aliases
   • Adds models specified in models that weren’t in the aliases map
   • Ensures all explicitly allowed models appear in the list
   • Uses the model name itself as the deployment value
   • Skips digit-only model IDs (reserved for custom models)

​

Model Filtering Logic

• If models is empty and no aliases are configured: No models are returned
• If models is empty but aliases are configured: Only aliased models are returned
• If models is ["*"]: All models from all three passes are included (unrestricted)
• If models is non-empty: Only models/aliases whose request names appear in models are included
• Duplicate Prevention: Each model ID is tracked to prevent duplicates across passes

​

Model Name Formatting

• gemini-pro → “Gemini Pro”
• claude-3-5-sonnet → “Claude 3 5 Sonnet”
• gemini_2_flash → “Gemini 2 Flash”

​

Example Configuration

{
  "aliases": {
    "my-gemini-ft": "1234567890",
    "my-claude-ft": "9876543210"
  },
  "vertex_key_config": {
    "project_id": "my-project",
    "region": "us-central1"
  }
}

{
  "aliases": {
    "gemini-2.0-flash": "gemini-2.0-flash",
    "claude-3-5-sonnet": "claude-3-5-sonnet-v2@20241022"
  },
  "vertex_key_config": {
    "project_id": "my-project",
    "region": "us-central1"
  }
}

{
  "models": ["gemini-2.0-flash", "claude-3-5-sonnet"],
  "aliases": {
    "gemini-2.0-flash": "gemini-2.0-flash",
    "claude-3-5-sonnet": "claude-3-5-sonnet-v2@20241022",
    "gemini-1.5-pro": "gemini-1.5-pro"
  },
  "vertex_key_config": {
    "project_id": "my-project",
    "region": "us-central1"
  }
}

​

Pagination

​

Caveats

​

Configuration

​

Video Generation