> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbifrost.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Baseten

> Baseten guide for Model APIs chat, streaming, tool calling, reasoning, the Responses API and decision models, plus dedicated deployments for chat, embeddings, transcription and image generation

## Overview

Baseten serves open models two ways, and the Bifrost `baseten` provider covers both:

* **Model APIs.** A shared, OpenAI-compatible endpoint at `https://inference.baseten.co/v1` for models such as `zai-org/GLM-5.3`, `deepseek-ai/DeepSeek-V4-Pro-0813` and `openai/gpt-oss-120b`. Bifrost uses it by default.
* **Dedicated deployments.** A model you deployed yourself, with its own URL such as `https://model-<id>.api.baseten.co/environments/production/sync/v1`. You point a model alias at it with `deployment_url`.

Both take your Baseten API key as a bearer token.

### Supported operations

| Operation | Model APIs | Dedicated deployment | Endpoint |
| - | - | - | - |
| Chat Completions | ✅ | ✅ | `/chat/completions` |
| Chat Completions (stream) | ✅ | ✅ | `/chat/completions` |
| Responses API | ✅ | ✅ (through chat) | `/responses` (Model APIs), `/chat/completions` (deployments) |
| Responses API (stream) | ✅ | ✅ (through chat) | `/responses` (Model APIs), `/chat/completions` (deployments) |
| Embeddings | ❌ | ✅ | `/embeddings` |
| Transcriptions (STT) | ❌ | ✅ | `/audio/transcriptions` |
| Rerank | ❌ | ✅ | `/sync/rerank` |
| Image Generation | ❌ | ✅ | `/images/generations`, or the deployment's `/predict` |
| Decisions | ✅ | ❌ | `/systemone` |
| List Models | ✅ | - | `/models` |
| Text Completions | ❌ | ❌ | - |
| Speech (TTS), Image Edit, Image Variation, Image streaming, Video | ❌ | ❌ | - |
| Files, Batch, Count Tokens | ❌ | ❌ | - |

Model APIs has no embeddings, rerank, audio or image routes, so those operations need a dedicated deployment. A request without one fails with an error that names the missing `deployment_url`. Baseten's TTS and classification deployments, and rerankers served on `/predict`, use their own request shapes, which Bifrost doesn't translate. FLUX.1 image Trusses are the exception: see [image generation](#5-image-generation). Baseten has no video generation models.

## Setup and configuration

> [tab] Web UI
>
> 1. Navigate to **Models** > **Model Providers**. If **Baseten** isn't under **Configured Providers**, click **Add New Provider** and select **Baseten**.
> 2. Click **Add Key** or edit an existing key.
> 3. Set a name for your key.
> 4. Paste your Baseten API key directly or use an environment variable (for example, `env.BASETEN_API_KEY`).
> 5. Save the provider configuration.

> [tab] config.json
>
> ```json theme={null}
> {
>   "providers": {
>     "baseten": {
>       "keys": [
>         {
>           "name": "baseten-key-1",
>           "value": "env.BASETEN_API_KEY",
>           "models": ["*"],
>           "weight": 1.0
>         }
>       ]
>     }
>   }
> }
> ```

> [tab] API
>
> Refer to the API documentation for [Provider Keys Management](https://docs.getbifrost.ai/api-reference/providers/create-a-key-for-a-provider.md).

> [tab] Go SDK
>
> ```go theme={null}
> case schemas.Baseten:
>     return []schemas.Key{{
>         Name:   "baseten-key-1",
>         Value:  *schemas.NewSecretVar("env.BASETEN_API_KEY"),
>         Models: []string{"*"},
>         Weight: 1.0,
>     }}, nil
> ```

`network_config.base_url` changes the Model APIs host. Leave it empty to use `https://inference.baseten.co/v1`.

## Dedicated deployments

Add an alias to the key and set its `deployment_url` to the deployment's OpenAI-compatible base URL. Find it on the deployment's page in the Baseten dashboard: it ends in `/sync/v1`. Requests for that alias go to the deployment instead of Model APIs, using the same API key.

The alias `model_id` is what Bifrost sends as `model`:

* **Engine-Builder (TensorRT-LLM) and BEI deployments** ignore `model`, so any value works. Baseten's examples use `not-required`.
* **vLLM and custom servers** need the name the server was started with (`--served-model-name`). Any other value returns a 404.

Set `model_name` to the model the deployment runs, so pricing and logs show a real model name instead of `not-required`.

> [tab] Web UI
>
> 1. Edit the Baseten key and open **Deployments**.
> 2. Add a deployment. The name is what clients send, for example `my-glm`, and the model ID is the served model name or `not-required`.
> 3. Expand the row and fill **Deployment URL** with the `/sync/v1` URL, or an env reference such as `env.BASETEN_GLM_DEPLOYMENT_URL`.
> 4. Save the key.
>
> The form rejects a literal URL that isn't `https`.

> [tab] config.json
>
> ```json theme={null}
> {
>   "providers": {
>     "baseten": {
>       "keys": [
>         {
>           "name": "baseten-key-1",
>           "value": "env.BASETEN_API_KEY",
>           "models": ["*"],
>           "weight": 1.0,
>           "aliases": {
>             "my-glm": {
>               "model_id": "not-required",
>               "model_name": "zai-org/GLM-5.3",
>               "deployment_url": "https://model-abc123.api.baseten.co/environments/production/sync/v1"
>             },
>             "my-embedder": {
>               "model_id": "not-required",
>               "model_name": "Qwen/Qwen3-Embedding-0.6B",
>               "deployment_url": "env.BASETEN_EMBEDDINGS_DEPLOYMENT_URL"
>             }
>           }
>         }
>       ]
>     }
>   }
> }
> ```

Clients then call the alias like any other model:

```bash theme={null}
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baseten/my-glm",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

A literal `deployment_url` must be an absolute `https` URL. Bifrost checks env and vault references when they resolve. If one resolves to an empty value, the request fails rather than falling back to Model APIs. `deployment_url` is a host Bifrost sends the key to, so it gets the same endpoint-change protection as Azure's per-alias `endpoint`.

To target a named environment or a specific deployment instead of production, use that deployment's own `/sync/v1` URL from the dashboard.

***

# 1. Chat Completions

## Request parameters

Baseten accepts the usual OpenAI chat parameters, including `tools`, `tool_choice`, `parallel_tool_calls`, `response_format` (`json_object` and `json_schema`), `logprobs`, `top_logprobs`, `seed`, `stop`, `temperature`, `top_p` and the penalties. For the full parameter reference, see [OpenAI Chat Completions](https://docs.getbifrost.ai/providers/supported-providers/openai.md#1-chat-completions).

Baseten's chat schema rejects fields it doesn't define, so Bifrost adjusts the request before sending it:

| Bifrost parameter | Sent to Baseten |
| - | - |
| `max_completion_tokens` | `max_tokens` |
| `service_tier`, `store`, `prompt_cache_key`, `verbosity` | dropped |
| `metadata`, `safety_identifier`, `audio`, `modalities`, `prediction`, `web_search_options` | dropped |
| Anthropic-only fields (`speed`, `inference_geo`, `mcp_servers`, `container`, `cache_control`, `task_budget`, `context_management`) | dropped |
| `developer` role | `system` role |

`n` and `best_of` only accept 1. Larger values come back as Baseten's own 400.

```bash theme={null}
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baseten/zai-org/GLM-5.3",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_completion_tokens": 512
  }'
```

### Reasoning

`reasoning_effort` passes through with every level Baseten accepts: `none`, `minimal`, `low`, `medium`, `high`, `xhigh` and `max`. Models without reasoning ignore it. Baseten returns the thinking as `reasoning_content`, which Bifrost exposes as the message's `reasoning`, and keeps it on assistant turns you send back. Reflection's Beam returns a 400 when earlier assistant turns lack it.

GLM 5.2 and Nemotron Ultra turn thinking on through the chat template instead of `reasoning_effort`. Send it as an extra param:

```json theme={null}
{
  "model": "baseten/zai-org/GLM-5.2",
  "messages": [{"role": "user", "content": "Plan a three-day trip to Kyoto."}],
  "chat_template_args": {"enable_thinking": true}
}
```

### Extra parameters

Extra parameters pass through unchanged, so Baseten-only sampling fields such as `top_k`, `min_p`, `repetition_penalty`, `min_tokens` and `chat_template_args` reach the API. Baseten's documented name is `chat_template_args`, not `chat_template_kwargs`.

### Streaming

Baseten sends usage on every streamed chunk by default. Bifrost reports it once, on the final chunk, with the cumulative totals.

### Prompt caching

Baseten caches prompt prefixes automatically. Sending the same `x-session-affinity` value on related requests improves hit rates. A client forwards it as `x-bf-eh-x-session-affinity`, or you can set it for every request in `network_config.extra_headers`.

***

# 2. Responses API

On Model APIs, Responses requests go to Baseten's native `/responses`, streamed or not. It is stateless: `previous_response_id`, `conversation`, `background` and `prompt` return 501, so send the full input on every turn.

Baseten's `/responses` accepts `reasoning.effort` from `minimal` to `high`, a narrower range than chat. Bifrost sends `none` as `minimal`, and `xhigh` or `max` as `high`.

Dedicated deployments have no `/responses` route. For an alias with `deployment_url`, Bifrost converts the request to chat completions and converts the reply back.

***

# 3. Embeddings

Embeddings run on dedicated BEI deployments, which serve the OpenAI embeddings shape.

```bash theme={null}
curl -X POST http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baseten/my-embedder",
    "input": ["first document", "second document"]
  }'
```

`my-embedder` must be an alias with a `deployment_url`. A request for a model without one, including any Model APIs model, fails before reaching Baseten.

## Rerank

Cross-encoder rerankers deployed on BEI with the `/rerank` route, such as the ModernBERT reranker in Baseten's BEI examples, serve it under the deployment's `/sync` base. Set the alias `deployment_url` to the deployment's `/sync/v1` URL, as for embeddings, and Bifrost calls `/sync/rerank` beside it.

```bash theme={null}
curl -X POST http://localhost:8080/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baseten/my-reranker",
    "query": "What causes tides?",
    "documents": ["The Moon pulls on the oceans.", "Clouds form from water vapor."],
    "top_n": 1,
    "return_documents": true
  }'
```

| Parameter | BEI handling |
| - | - |
| `query` | `query` |
| `documents` | `texts`, each document's `text` in order |
| `top_n` | applied by Bifrost to the sorted results, since BEI scores every document |
| `return_documents` | applied by Bifrost, attaching each request document by `index` |
| `max_tokens_per_doc`, `priority` | dropped |

Results come back highest `relevance_score` first, with each document's `index` and `id`. BEI fields such as `raw_scores`, `truncate` and `truncation_direction` pass through as extra params with `x-bf-passthrough-extra-params: true`.

Baseten's Qwen3 Reranker and zerank recipes are deployed with `/predict` as their route and take model-specific prompts in `inputs`. Bifrost doesn't build those prompts, so use a `/rerank` cross-encoder for `/v1/rerank`.

***

# 4. Transcriptions

Dedicated speech-recognition deployments (for example Qwen3 ASR) serve OpenAI's multipart `/audio/transcriptions`, so `/v1/audio/transcriptions` works against an alias with `deployment_url`. Streaming transcription isn't supported, because Baseten's real-time transcription uses its own WebSocket protocol.

***

# 5. Image Generation

Baseten's image recipes, such as FLUX.2, Krea 2 Turbo and Qwen-Image, run on the SGLang diffusion stack and serve OpenAI's `/images/generations`, so `/v1/images/generations` works against an alias with `deployment_url`.

```json theme={null}
{
  "providers": {
    "baseten": {
      "keys": [
        {
          "name": "baseten-key-1",
          "value": "env.BASETEN_API_KEY",
          "models": ["*"],
          "weight": 1.0,
          "aliases": {
            "my-flux": {
              "model_id": "black-forest-labs/FLUX.2-dev",
              "model_name": "black-forest-labs/FLUX.2-dev",
              "deployment_url": "https://model-def456.api.baseten.co/environments/production/sync/v1"
            }
          }
        }
      ]
    }
  }
}
```

```bash theme={null}
curl -X POST http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baseten/my-flux",
    "prompt": "A lighthouse at dusk",
    "size": "1024x1024",
    "response_format": "b64_json"
  }'
```

Images come back as base64 in `data[].b64_json`. The deployments can't return image URLs, so when a request sets no `response_format`, Bifrost sends `b64_json` instead of leaving OpenAI's `url` default in place. Streaming, edits and variations aren't supported, because the deployments don't serve those routes. A request for a model without a `deployment_url`, including any Model APIs model, fails before reaching Baseten.

## Using the /predict URL from the dashboard

The Baseten dashboard shows each deployment's `/predict` URL, for example `https://model-def456.api.baseten.co/environments/production/predict`. You can use it as `deployment_url` for image generation. Bifrost posts the same OpenAI image body to that URL as given, without appending `/images/generations`:

* FLUX.2, Krea 2 Turbo and Qwen-Image map `/predict` to their `/v1/images/generations` route, so they behave as with the `/sync/v1` URL.
* FLUX.1 dev and schnell are packaged as a Truss that only serves `/predict`. It reads `prompt`, `seed` and `num_inference_steps` from the body, ignores the other fields, and returns one base64 image. `size` has no effect, so pass `width` and `height` as extra params with `x-bf-passthrough-extra-params: true` for anything other than 1024x1024.

Bifrost reads both reply shapes, the Truss's `{"data": "<base64>"}` and OpenAI's `data[].b64_json`, into `data[].b64_json`. A reply with no image fails rather than returning an empty list. A custom Truss with a different request or reply shape isn't supported.

A `/predict` URL only works for image generation. Chat, embeddings and transcription against that alias fail before any call, with an error that asks for the deployment's `/sync/v1` URL.

***

# 6. Decisions

Baseten's decision models answer [`/v1/decisions`](https://docs.getbifrost.ai/quickstart/gateway/decisions) through Model APIs' System One endpoint, `/systemone`. They classify, route and score an input with typed answers instead of generated text.

| Model | Slug | Context limit |
| - | - | - |
| joey-1 | `decision-models/baseten-joey-2b-latest` | 16,384 tokens |
| elephant-1 | `decision-models/baseten-elephant-9b-latest` | 16,384 tokens |
| mammoth-1 | `decision-models/baseten-mammoth-27b-latest` | 16,384 tokens |

```bash theme={null}
curl -X POST http://localhost:8080/v1/decisions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "baseten/decision-models/baseten-joey-2b-latest",
    "input": "I was charged twice. Refund me today or I cancel.",
    "questions": [
      {"type": "predicate", "name": "wants_refund", "instructions": "Is the customer asking for a refund?"},
      {"type": "choice", "name": "route", "choices": [
        {"value": "billing", "description": "Payments and refunds"},
        {"value": "technical_support", "description": "Software defects"}
      ]}
    ]
  }'
```

How Bifrost maps the request:

* `predicate`, `choice` and `score` questions become System One's `noul`, `choice` and `score`. `box` questions aren't supported and come back as Baseten's 400. A request can have at most 255 choices per question, 2 to 10 score levels, and 512 options in total, where each predicate counts as two.
* Text and structured input are sent as the `state` unchanged. A message input is sent as `{"messages": [...]}`, because Baseten reads a top-level array as data rather than a conversation. Only `user` and `assistant` roles are accepted, and a message without a role is sent as `user`. Any other role fails before reaching Baseten.
* Images go inline as `input_image` parts on user messages, as a base64 PNG, JPEG or WebP data URL or an HTTPS URL on Amazon S3, Google Cloud Storage or Azure Blob Storage, up to 16 per request. An image `detail` other than `auto` is dropped, since Baseten accepts only `auto`. Images on assistant messages, more than 16 images, images sent by reference (with an `id`), and tool calls or outputs in the input fail before reaching Baseten. Bifrost passes images to every decision model, but Baseten trained elephant-1 and mammoth-1 on text only, so they may not read an image reliably.
* Baseten's `service_tier` (`flex` or `priority`) and `max_len` are sent when the request carries them as extra params with `x-bf-passthrough-extra-params: true`.

Bifrost sends Decisions to Model APIs and ignores an alias `deployment_url`, even though Baseten also offers decision models as dedicated deployments. With the [compat plugin's](https://docs.getbifrost.ai/features/compat-plugin.md) `convert_decision_to_responses` on, `decision-models/` models still go to Baseten's endpoint, and any other Baseten model is emulated through the Responses API.

***

# 7. List Models

Bifrost lists models from Model APIs' `/models` for each key, plus the aliases configured on the key. Dedicated deployments appear through their aliases.

***

# 8. Errors and retries

Baseten returns 529 when a model is overloaded. Bifrost treats it as transient, so it is retried with the provider's backoff settings when `network_config.max_retries` is above 0. `max_retries` defaults to 0, so set it before relying on retries; see [Retries and fallbacks](https://docs.getbifrost.ai/features/retries-and-fallbacks.md). Other errors keep their status code and message.

***

## Baseten through Hugging Face

Hugging Face's inference router also reaches Baseten-hosted models, as `huggingface/baseten/<model>`. That path uses your Hugging Face token and Hugging Face's model IDs. The `baseten` provider calls Baseten directly with a Baseten API key, and it is the only way to reach your dedicated deployments.

If you set up Baseten earlier as a custom provider named `baseten`, that name now belongs to this provider. The dashboard offers to rename the custom one so both can exist side by side.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.