Overview

SGL (SGLang) is an OpenAI-compatible local/remote inference engine used for serving models with high throughput. By default Bifrost delegates operations to the OpenAI provider implementation; Chat Completions and the Responses API can instead be routed through SGLang’s Anthropic-compatible Messages endpoint. Key features:

OpenAI API compatibility - Identical request/response format
Optional Anthropic-compatible mode - Route chat and responses through /v1/messages with use_anthropic_endpoints
Full streaming support - Server-Sent Events with usage tracking
Tool calling - Complete function definition and execution
Text embeddings - Support for embedding models
Parameter filtering - Removes unsupported fields for compatibility

Supported Operations

Operation	Non-Streaming	Streaming	Endpoint (default)	Endpoint (`use_anthropic_endpoints: true`)
Chat Completions	✅	✅	`/v1/chat/completions`	`/v1/messages`
Responses API	✅	✅	`/v1/chat/completions`	`/v1/messages`
Text Completions	✅	✅	`/v1/completions`	`/v1/completions` (unaffected)
Embeddings	✅	-	`/v1/embeddings`	`/v1/embeddings` (unaffected)
List Models	✅	-	`/v1/models`	`/v1/models` (unaffected)
Count Tokens	✅	-	`/v1/messages/count_tokens`	`/v1/messages/count_tokens` (always)
Image Generation	❌	❌	-	-
Speech (TTS)	❌	❌	-	-
Transcriptions (STT)	❌	❌	-	-
Files	❌	❌	-	-
Batch	❌	❌	-	-

Unsupported Operations (❌): Speech, Transcriptions, Files, and Batch are not supported by the upstream SGL API. These return UnsupportedOperationError.SGL is typically self-hosted. Ensure BaseURL is configured correctly pointing to your SGL instance (e.g., http://localhost:8000).

Setup & Configuration

Configure SGLang as a provider.

Web UI
config.json
API
Go SDK

Navigate to Models > Model Providers. Look for SGLang under Configured Providers. If it is missing, click on Add New Provider and select SGLang.
Click Add New Server or edit an existing key.
Set a name for your key.
Leave API Key blank for local servers. If your endpoint requires auth, paste a bearer token directly or use an environment variable.
Set SGLang URL to http://localhost:8000 or your remote SGLang endpoint.
Set Allowed Models to All Models (default) or the specific model allowlist you want this key to serve.
Save the provider configuration.

{
  "providers": {
    "sgl": {
      "keys": [
        {
          "name": "sgl-local",
          "value": "",
          "models": [
            "*"
          ],
          "weight": 1.0,
          "sgl_key_config": {
            "url": "http://localhost:8000"
          }
        }
      ]
    }
  }
}

case schemas.SGL:
    return []schemas.Key{{
        Name:   "sgl-local",
        Value:  *schemas.NewSecretVar(""),
        Models: []string{"*"},
        Weight: 1.0,
        SGLKeyConfig: &schemas.SGLKeyConfig{
            URL: *schemas.NewSecretVar("http://localhost:8000"),
        },
    }}, nil

Anthropic-Compatible Endpoints (optional)

SGLang can serve an Anthropic-compatible Messages endpoint (/v1/messages) alongside its OpenAI-compatible APIs. Setting use_anthropic_endpoints routes Chat Completions and the Responses API through that endpoint instead. Text Completions and Embeddings are unaffected and always use their OpenAI-compatible endpoints. Two details specific to this mode:

Authentication does not change. Bifrost sends Authorization: Bearer <key> either way, and omits the header when the key value is empty. It additionally sends anthropic-version: 2023-06-01.
Count Tokens is not governed by this setting. It always uses /v1/messages/count_tokens.

The setting can be configured per key, and overridden per model alias:

Key-level - Sets the default endpoint mode for every request made with that key.
Alias-level - Overrides the key-level default for a single alias, so one key can serve some aliases through the OpenAI-compatible endpoints and others through the Anthropic-compatible endpoint.

If neither is set, requests fall back to SGLang’s OpenAI-compatible endpoints.

Web UI
API
config.json

On the key form, toggle Use Anthropic Endpoints (off by default). To override this for a specific alias, open that alias’s expanded row in the deployments table and toggle Use Anthropic endpoints under SGLang overrides, which takes priority over the key-level setting for that alias only.

The use_anthropic_endpoints boolean is part of the same key payload used by Provider Keys Management, and the alias payload for that key’s models entries.

{
  "providers": {
    "sgl": {
      "keys": [
        {
          "name": "sgl-local",
          "value": "",
          "models": [
            "*"
          ],
          "weight": 1.0,
          "use_anthropic_endpoints": true,
          "sgl_key_config": {
            "url": "http://localhost:8000"
          }
        }
      ]
    }
  }
}

To override this per-alias (for example, on a virtual key’s model config), set use_anthropic_endpoints alongside the alias’s model_id:

{
  "model_id": "meta-llama/Llama-3.2-1B-Instruct",
  "use_anthropic_endpoints": false
}

Field	Type	Required	Description
`use_anthropic_endpoints`	boolean	No	Routes chat completions and responses requests through Anthropic-compatible endpoints. Default: `false`.

Anthropic’s server and client tools (web_search, web_fetch, code_execution, computer, bash, memory, text_editor, tool_search, mcp_toolset) run on Anthropic-operated infrastructure, so a self-hosted SGLang server does not implement them. Bifrost drops them from the request rather than forwarding a tool the server will reject. Your own function tools are never affected.This matters most for clients that enable a built-in web search by default, which would otherwise fail every request.

1. Chat Completions

Request Parameters

SGL supports all standard OpenAI chat completion parameters. For full parameter reference and behavior, see OpenAI Chat Completions.

Filtered Parameters

Removed for SGL compatibility:

prompt_cache_key - Not supported
verbosity - Anthropic-specific
store - Not supported
service_tier - OpenAI-specific

SGL supports all standard OpenAI message types, tools, responses, and streaming formats. For details on message handling, tool conversion, responses, and streaming, refer to OpenAI Chat Completions.

2. Responses API

By default, Responses fall back to Chat Completions with format conversion:

ResponsesRequest → ChatRequest → Response conversion

Same parameter support as Chat Completions. With use_anthropic_endpoints enabled on the key or alias, Responses skip that fallback and are converted to the Anthropic Messages format, then sent natively to /v1/messages. See Anthropic-Compatible Endpoints.

3. Text Completions

SGL supports legacy text completion format:

Parameter	Mapping
`prompt`	Direct pass-through
`max_tokens`	max_tokens
`temperature`, `top_p`	Direct pass-through
`frequency_penalty`, `presence_penalty`	Supported

4. Embeddings

SGL supports text embeddings for vector generation:

Parameter	Notes
`input`	Text or array of texts
`model`	Embedding model name
`encoding_format`	”float” or “base64”
`dimensions`	Model-specific dimension count

Response returns embedding vectors with usage information.

5. List Models

Lists available models from SGL server with capabilities.

Unsupported Features

Feature	Reason
Speech/TTS	Not offered by SGL API
Transcription/STT	Not offered by SGL API
Batch Operations	Not offered by SGL API
File Management	Not offered by SGL API

SGL requires BaseURL configuration pointing to your SGL instance (e.g., http://localhost:8000 for local, https://sgl.example.com for remote).

Caveats

BaseURL Configuration Required

Severity: High Behavior: BaseURL must be explicitly configured through sgl_key_config.url or network_config.base_url Impact: Requests fail without proper configuration Code: Requests call baseURLOrError before contacting SGL

Cache Control Stripped

Severity: Medium Behavior: Cache control directives are removed from messages Impact: Prompt caching features don’t work Code: Stripped during JSON marshaling

Parameter Filtering

Severity: Low Behavior: OpenAI-specific fields filtered out Impact: prompt_cache_key, verbosity, store removed Code: filterOpenAISpecificParameters

User Field Size Limit

Severity: Low Behavior: User field > 64 characters silently dropped Impact: Longer user identifiers are lost Code: SanitizeUserField enforces 64-char max

Overview

Quick Start

Release Cadence

Migration Guides

Migrate from other Gateway

SDK Integrations

Providers & Guides

MCP Gateway

Custom plugins

Open Source Features

SGLang

Overview

Supported Operations

Setup & Configuration

Anthropic-Compatible Endpoints (optional)

1. Chat Completions

Request Parameters

Filtered Parameters

2. Responses API

3. Text Completions

4. Embeddings

5. List Models

Unsupported Features

Caveats

​Overview

​Supported Operations

​Setup & Configuration

​Anthropic-Compatible Endpoints (optional)

​1. Chat Completions

​Request Parameters

​Filtered Parameters

​2. Responses API

​3. Text Completions

​4. Embeddings

​5. List Models

​Unsupported Features

​Caveats

Overview

Supported Operations

Setup & Configuration

Anthropic-Compatible Endpoints (optional)

1. Chat Completions

Request Parameters

Filtered Parameters

2. Responses API

3. Text Completions

4. Embeddings

5. List Models

Unsupported Features

Caveats