​

Overview

• Role remapping - “assistant” → “model”, system messages integrated into main flow
• Message grouping - Consecutive tool responses merged into single user message
• Parameter renaming - e.g., max_completion_tokens → maxOutputTokens, stop → stopSequences
• Function call handling - Tool call ID preservation and thought signature support
• Content modality - Support for text, images, video, code execution, and thought content
• Thinking/Reasoning - Thinking configuration mapped to Bifrost reasoning structure

​

Supported Operations

​

Authentication

​

API Key Authentication

1. Header Method (standard Gemini endpoints):
   • Format: x-goog-api-key: YOUR_API_KEY header
   • Used for: Standard Gemini endpoints (e.g., /v1beta/models/{model}:generateContent)
2. Query Parameter Method (Imagen and custom endpoints):
   • Format: ?key=YOUR_API_KEY appended to request URLs
   • Used for: Imagen models and custom endpoints
   • Example: https://generativelanguage.googleapis.com/v1beta/models/imagen-4.0-generate-001:predict?key=YOUR_API_KEY

​

1. Chat Completions

​

Request Parameters

​

Parameter Mapping

​

Dropped Parameters

​

Extra Parameters

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini/gemini-2.0-flash",
    "messages": [{"role": "user", "content": "Hello"}],
    "top_k": 40,
    "stop_sequences": ["###"]
  }'

resp, err := client.ChatCompletionRequest(ctx, &schemas.BifrostChatRequest{
    Provider: schemas.Gemini,
    Model:    "gemini-2.0-flash",
    Input:    messages,
    Params: &schemas.ChatParameters{
        ExtraParams: map[string]interface{}{
            "top_k": 40,
            "stop_sequences": []string{"###"},
        },
    },
})

​

Reasoning / Thinking

​

Parameter Mapping

• reasoning.effort → thinkingConfig.thinkingLevel (“low” → LOW, “high” → HIGH)
• reasoning.max_tokens → thinkingConfig.thinkingBudget (token budget for thinking)
• reasoning parameter triggers thinkingConfig.includeThoughts = true

​

Supported Thinking Levels

• "low" / "minimal" → LOW
• "medium" / "high" → HIGH
• null or unspecified → Based on max_tokens: -1 (dynamic), 0 (disabled), or specific budget

​

Example

// Request
{"reasoning": {"effort": "high", "max_tokens": 10000}}

// Gemini conversion
{"thinkingConfig": {"includeThoughts": true, "thinkingLevel": "HIGH", "thinkingBudget": 10000}}

​

Message Conversion

​

Critical Caveats

• Role remapping: “assistant” → “model”, “system” → part of user/model content flow
• Consecutive tool responses: Tool response messages merged into single user message with function response parts
• Content flattening: Multi-part content in single message preserved as parts array

​

Image Conversion

• URL images: {type: "image_url", image_url: {url: "..."}} → {type: "image", source: {type: "url", url: "..."}}
• Base64 images: Data URL → {type: "image", source: {type: "base64", media_type: "image/png", ...}}
• Video content: Preserved with metadata (fps, start/end offset)

​

Tool Conversion

• function.name → functionDeclarations.name (preserved)
• function.parameters → functionDeclarations.parameters (Schema format)
• function.description → functionDeclarations.description
• function.strict → Dropped (not supported by Gemini)

​

Tool Choice Mapping

​

Response Conversion

​

Field Mapping

• finishReason → finish_reason:
  • STOP → stop
  • MAX_TOKENS → length
  • SAFETY, RECITATION, LANGUAGE, BLOCKLIST, PROHIBITED_CONTENT, SPII, IMAGE_SAFETY → content_filter
  • MALFORMED_FUNCTION_CALL, UNEXPECTED_TOOL_CALL → tool_calls
• candidates[0].content.parts[0].text → choices[0].message.content (if single text block)
• candidates[0].content.parts[].functionCall → choices[0].message.tool_calls
• promptTokenCount → usage.prompt_tokens
• candidatesTokenCount → usage.completion_tokens
• totalTokenCount → usage.total_tokens
• cachedContentTokenCount → usage.prompt_tokens_details.cached_tokens
• thoughtsTokenCount → usage.completion_tokens_details.reasoning_tokens
• Thought content (from text parts with thought: true) → reasoning field in stream deltas
• Function call args (map) → JSON string arguments

​

Streaming

• Streaming responses contain deltas in delta.content (text), delta.reasoning (thoughts), delta.toolCalls (function calls)
• Function responses appear as text content in the delta
• finish_reason only set on final chunk
• Usage metadata only included in final chunk

​

2. Responses API

​

Request Parameters

​

Parameter Mapping

​

Extra Parameters

curl -X POST http://localhost:8080/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini/gemini-2.0-flash",
    "input": "Hello, how are you?",
    "instructions": "You are a helpful assistant.",
    "top_k": 40
  }'

resp, err := client.ResponsesRequest(ctx, &schemas.BifrostResponsesRequest{
    Provider: schemas.Gemini,
    Model:    "gemini-2.0-flash",
    Input:    messages,
    Params: &schemas.ResponsesParameters{
        Instructions: schemas.Ptr("You are a helpful assistant."),
        ExtraParams: map[string]interface{}{
            "top_k": 40,
        },
    },
})

​

Input & Instructions

• Input: String wrapped as user message or array converted to messages
• Instructions: Becomes system instruction (single text block)

​

Tool Support

• Computer tools auto-configured (if specified in Bifrost request)
• Function-based tools always enabled

​

Response Conversion

• finishReason → status: STOP/MAX_TOKENS/other → completed | SAFETY → incomplete
• Output items conversion:
  • Text parts → message field
  • Function calls → function_call field
  • Thought content → reasoning field
• Usage fields preserved with cache tokens mapped to *_tokens_details.cached_tokens

​

Streaming

• content_part.added emitted for text and reasoning parts
• Item IDs generated as msg_{responseID}_item_{outputIndex}

​

3. Speech (Text-to-Speech)

​

Request Parameters

​

Voice Configuration

{
  "generationConfig": {
    "responseModalities": ["AUDIO"],
    "speechConfig": {
      "voiceConfig": {
        "prebuiltVoiceConfig": {
          "voiceName": "Chant-Female"
        }
      }
    }
  }
}

{
  "generationConfig": {
    "responseModalities": ["AUDIO"],
    "speechConfig": {
      "multiSpeakerVoiceConfig": {
        "speakerVoiceConfigs": [
          {
            "speaker": "Character A",
            "voiceConfig": {
              "prebuiltVoiceConfig": {
                "voiceName": "Chant-Female"
              }
            }
          }
        ]
      }
    }
  }
}

​

Response Conversion

• Audio data extracted from candidates[0].content.parts[].inlineData
• Format conversion: Gemini returns PCM audio (s16le, 24kHz, mono)
• Auto-conversion: PCM → WAV when response_format: "wav" (default)
• Raw audio returned if response_format is omitted or empty string

​

Supported Voices

• Chant-Female - Female voice
• Chant-Male - Male voice
• Additional voices depend on model capabilities

​

4. Transcriptions (Speech-to-Text)

​

Request Parameters

​

Audio Input Handling

{
  "contents": [
    {
      "parts": [
        {
          "text": "<prompt text>"
        },
        {
          "inlineData": {
            "mimeType": "audio/wav",
            "data": "<base64-encoded-audio>"
          }
        }
      ]
    }
  ]
}

​

Extra Parameters

curl -X POST http://localhost:8080/v1/audio/transcriptions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini/gemini-2.0-flash",
    "file": "<binary-audio-data>",
    "prompt": "Transcribe this audio in the original language."
  }'

resp, err := client.TranscriptionRequest(ctx, &schemas.BifrostTranscriptionRequest{
    Provider: schemas.Gemini,
    Model:    "gemini-2.0-flash",
    Input: &schemas.TranscriptionInput{
        File: audioBytes,
    },
    Params: &schemas.TranscriptionParameters{
        Prompt: schemas.Ptr("Transcribe this audio."),
        ExtraParams: map[string]interface{}{
            "safety_settings": [...],
        },
    },
})

​

Response Conversion

• Transcribed text extracted from candidates[0].content.parts[].text
• task set to "transcribe"
• Usage metadata mapped:
  • promptTokenCount → input_tokens
  • candidatesTokenCount → output_tokens
  • totalTokenCount → total_tokens

​

5. Embeddings

• input → requests[0].content.parts[0].text (single text joins arrays with space)
• dimensions → outputDimensionality
• Extra task type and title via extra_params

• embeddings[].values → Bifrost embedding array
• metadata.billableCharacterCount → Usage prompt tokens (fallback)
• Token counts extracted from usage metadata

​

6. Batch API

• POST /v1beta/batchJobs - Create
• GET /v1beta/batchJobs?pageSize={limit}&pageToken={token} - List
• GET /v1beta/batchJobs/{batch_id} - Retrieve
• POST /v1beta/batchJobs/{batch_id}:cancel - Cancel

• Status mapping: BATCH_STATE_PENDING/BATCH_STATE_RUNNING → in_progress, BATCH_STATE_SUCCEEDED → completed, BATCH_STATE_FAILED → failed, BATCH_STATE_CANCELLING → cancelling, BATCH_STATE_CANCELLED → cancelled, BATCH_STATE_EXPIRED → expired
• Inline responses: Array in dest.inlinedResponses
• File-based responses: JSONL file in dest.fileName

​

7. Files API

• name → id
• displayName → filename
• sizeBytes → size_bytes
• mimeType → content_type
• createTime (RFC3339) → Converted to Unix timestamp

• POST /upload/storage/v1beta/files - Upload
• GET /v1beta/files?limit={limit}&pageToken={token} (cursor pagination)
• GET /v1beta/files/{file_id} - Retrieve
• DELETE /v1beta/files/{file_id} - Delete
• GET /v1beta/files/{file_id}/content - Download

​

8. Image Generation

1. Standard Gemini Format: Uses the /v1beta/models/{model}:generateContent endpoint
2. Imagen Format: Uses the /v1beta/models/{model}:predict endpoint for Imagen models (detected automatically)

​

Parameter Mapping

​

Extra Parameters

curl -X POST http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini/imagen-4.0-generate-001",
    "prompt": "A sunset over the mountains",
    "size": "1024x1024",
    "n": 2,
    "output_format": "png"
  }'

resp, err := client.ImageGenerationRequest(ctx, &schemas.BifrostImageGenerationRequest{
    Provider: schemas.Gemini,
    Model:    "imagen-4.0-generate-001",
    Input: &schemas.ImageGenerationInput{
        Prompt: "A sunset over the mountains",
    },
    Params: &schemas.ImageGenerationParameters{
        Size:         schemas.Ptr("1024x1024"),
        N:            schemas.Ptr(2),
        OutputFormat: schemas.Ptr("png"),
    },
})

​

Request Conversion

​

Standard Gemini Format

• Model mapping: bifrostReq.Model → req.Model, with bifrostReq.Input.Prompt → req.Contents[0].Parts[0].Text
• Response modality: Set by bifrost internally to generationConfig.responseModalities = ["IMAGE"] to indicate image generation
• Image count: Specify number of images via n → generationConfig.candidateCount
• Extra parameters: Include safetySettings, cachedContent, and labels mapped directly

​

Imagen Format

• Prompt: bifrostReq.Prompt → req.Instances[0].Prompt
• Number of Images: n → req.Parameters.SampleCount
• Size Conversion: size (WxH format) converted to:
  • imageSize: "1k" (if dimensions ≤ 1024), "2k" (if dimensions ≤ 2048). Sizes larger than "2k" are not supported by Imagen models.
  • aspectRatio: "1:1", "3:4", "4:3", "9:16", or "16:9" (based on width/height ratio)
• Output Format: output_format ("png", "jpeg") → parameters.outputOptions.mimeType ("image/png", "image/jpeg")
• Seed & Negative Prompt: Passed directly to seed and parameters.negativePrompt
• Extra Parameters: personGeneration, language, enhancePrompt, safetySettings mapped to parameters

​

Response Conversion

​

Standard Gemini Format

• Image Data: Extracts InlineData from candidates[0].content.parts[] with MIME type image/*
• Output Format: Converts MIME type (image/png, image/jpeg, image/webp) → file extension (png, jpeg, webp)
• Usage: Extracts token usage from usageMetadata
• Multiple Images: Each image part becomes an ImageData entry in the response array

​

Imagen Format

• Image Data: Each prediction in response.predictions[] → ImageData with b64_json from bytesBase64Encoded
• Output Format: Converts prediction.mimeType → file extension for outputFormat field (Imagen doesnt support webp)
• Index: Each prediction gets an index (0, 1, 2, …) in the response array

​

Size Conversion

​

Endpoint Selection

• Imagen models (detected via schemas.IsImagenModel()): Uses /v1beta/models/{model}:predict endpoint
• Other models: Uses /v1beta/models/{model}:generateContent endpoint with image response modality

​

Streaming

​

9. Image Edit

1. Standard Gemini Format: Uses the /v1beta/models/{model}:generateContent endpoint (for Gemini models)
2. Imagen Format: Uses the /v1beta/models/{model}:predict endpoint (for Imagen models, detected automatically)

​

Standard Gemini Format (Non-Imagen Models)

• Model & Prompt: bifrostReq.Model → req.Model, bifrostReq.Input.Prompt → req.Contents[0].Parts[0].Text
• Images: Each image in bifrostReq.Input.Images is converted to a Part with:
  • MIME type detection (image/jpeg, image/webp, image/png) with fallback to image/png
  • Base64 encoding: image.Image → Part.InlineData.Data (base64 string)
  • MIME type: Part.InlineData.MIMEType
• Response Modality: GenerationConfig.ResponseModalities is set to [ModalityImage] to indicate image generation
• Extra Parameters: Extracted from ExtraParams:
  • safetySettings / safety_settings → SafetySettings
  • cachedContent / cached_content → CachedContent
  • labels → Labels (map[string]string)

​

Imagen Format (Imagen Models)

• Reference Images: Each image in bifrostReq.Input.Images is converted to ReferenceImage with:
  • ReferenceType: "REFERENCE_TYPE_RAW"
  • ReferenceID: Sequential IDs starting from 1
  • ReferenceImage.BytesBase64Encoded: Base64-encoded image data
• Mask Configuration: If Params.Mask is provided or maskMode is specified:
  • Default maskMode: "MASK_MODE_USER_PROVIDED" when mask data is present
  • maskMode can be overridden via ExtraParams["maskMode"]
  • dilation extracted from ExtraParams["dilation"] (validated to range [0, 1])
  • maskClasses extracted from ExtraParams["maskClasses"] (for MASK_MODE_SEMANTIC)
  • Mask image (if provided) is base64-encoded and added as ReferenceType: "REFERENCE_TYPE_MASK"
• Edit Mode Mapping: Params.Type is mapped to EditMode:
  • "inpainting" → "EDIT_MODE_INPAINT_INSERTION"
  • "outpainting" → "EDIT_MODE_OUTPAINT"
  • "inpaint_removal" → "EDIT_MODE_INPAINT_REMOVAL"
  • "bgswap" → "EDIT_MODE_BGSWAP"
  • If Type is not set, editMode can be specified directly via ExtraParams["editMode"]
• Parameters:
  • n → Parameters.SampleCount
  • output_format → Parameters.OutputOptions.MimeType (converted: "png" → "image/png", etc.)
  • output_compression → Parameters.OutputOptions.CompressionQuality
  • seed (via ExtraParams["seed"]) → Parameters.Seed
  • negativePrompt (via ExtraParams["negativePrompt"]) → Parameters.NegativePrompt
  • guidanceScale (via ExtraParams["guidanceScale"]) → Parameters.GuidanceScale
  • baseSteps (via ExtraParams["baseSteps"]) → Parameters.BaseSteps
  • Additional Imagen-specific parameters: addWatermark, includeRaiReason, includeSafetyAttributes, personGeneration, safetySetting, language, storageUri

• Standard Gemini Format: Uses the same response conversion as image generation (see Image Generation section)
• Imagen Format: Uses the same response conversion as Imagen image generation (see Image Generation section)

• Imagen models (detected via schemas.IsImagenModel()): Uses /v1beta/models/{model}:predict endpoint
• Other models: Uses /v1beta/models/{model}:generateContent endpoint with image response modality

​

10. List Models

• name (remove “models/” prefix) → id (add “gemini/” prefix)
• displayName → name
• description → description
• inputTokenLimit → max_input_tokens
• outputTokenLimit → max_output_tokens
• Context length = inputTokenLimit + outputTokenLimit

​

Content Type Support

​

Caveats