Skip to main content

Overview

Baseten serves open models two ways, and the Bifrost baseten provider covers both:
  • Model APIs. A shared, OpenAI-compatible endpoint at https://inference.baseten.co/v1 for models such as zai-org/GLM-5.3, deepseek-ai/DeepSeek-V4-Pro-0813 and openai/gpt-oss-120b. Bifrost uses it by default.
  • Dedicated deployments. A model you deployed yourself, with its own URL such as https://model-<id>.api.baseten.co/environments/production/sync/v1. You point a model alias at it with deployment_url.
Both take your Baseten API key as a bearer token.

Supported operations

Model APIs has no embeddings, rerank, audio or image routes, so those operations need a dedicated deployment. A request without one fails with an error that names the missing deployment_url. Baseten’s TTS and classification deployments, and rerankers served on /predict, use their own request shapes, which Bifrost doesn’t translate. FLUX.1 image Trusses are the exception: see image generation. Baseten has no video generation models.

Setup and configuration

  1. Navigate to Models > Model Providers. If Baseten isn’t under Configured Providers, click Add New Provider and select Baseten.
  2. Click Add Key or edit an existing key.
  3. Set a name for your key.
  4. Paste your Baseten API key directly or use an environment variable (for example, env.BASETEN_API_KEY).
  5. Save the provider configuration.
network_config.base_url changes the Model APIs host. Leave it empty to use https://inference.baseten.co/v1.

Dedicated deployments

Add an alias to the key and set its deployment_url to the deployment’s OpenAI-compatible base URL. Find it on the deployment’s page in the Baseten dashboard: it ends in /sync/v1. Requests for that alias go to the deployment instead of Model APIs, using the same API key. The alias model_id is what Bifrost sends as model:
  • Engine-Builder (TensorRT-LLM) and BEI deployments ignore model, so any value works. Baseten’s examples use not-required.
  • vLLM and custom servers need the name the server was started with (--served-model-name). Any other value returns a 404.
Set model_name to the model the deployment runs, so pricing and logs show a real model name instead of not-required.
  1. Edit the Baseten key and open Deployments.
  2. Add a deployment. The name is what clients send, for example my-glm, and the model ID is the served model name or not-required.
  3. Expand the row and fill Deployment URL with the /sync/v1 URL, or an env reference such as env.BASETEN_GLM_DEPLOYMENT_URL.
  4. Save the key.
The form rejects a literal URL that isn’t https.
Clients then call the alias like any other model:
A literal deployment_url must be an absolute https URL. Bifrost checks env and vault references when they resolve. If one resolves to an empty value, the request fails rather than falling back to Model APIs. deployment_url is a host Bifrost sends the key to, so it gets the same endpoint-change protection as Azure’s per-alias endpoint. To target a named environment or a specific deployment instead of production, use that deployment’s own /sync/v1 URL from the dashboard.

1. Chat Completions

Request parameters

Baseten accepts the usual OpenAI chat parameters, including tools, tool_choice, parallel_tool_calls, response_format (json_object and json_schema), logprobs, top_logprobs, seed, stop, temperature, top_p and the penalties. For the full parameter reference, see OpenAI Chat Completions. Baseten’s chat schema rejects fields it doesn’t define, so Bifrost adjusts the request before sending it: n and best_of only accept 1. Larger values come back as Baseten’s own 400.

Reasoning

reasoning_effort passes through with every level Baseten accepts: none, minimal, low, medium, high, xhigh and max. Models without reasoning ignore it. Baseten returns the thinking as reasoning_content, which Bifrost exposes as the message’s reasoning, and keeps it on assistant turns you send back. Reflection’s Beam returns a 400 when earlier assistant turns lack it. GLM 5.2 and Nemotron Ultra turn thinking on through the chat template instead of reasoning_effort. Send it as an extra param:

Extra parameters

Extra parameters pass through unchanged, so Baseten-only sampling fields such as top_k, min_p, repetition_penalty, min_tokens and chat_template_args reach the API. Baseten’s documented name is chat_template_args, not chat_template_kwargs.

Streaming

Baseten sends usage on every streamed chunk by default. Bifrost reports it once, on the final chunk, with the cumulative totals.

Prompt caching

Baseten caches prompt prefixes automatically. Sending the same x-session-affinity value on related requests improves hit rates. A client forwards it as x-bf-eh-x-session-affinity, or you can set it for every request in network_config.extra_headers.

2. Responses API

On Model APIs, Responses requests go to Baseten’s native /responses, streamed or not. It is stateless: previous_response_id, conversation, background and prompt return 501, so send the full input on every turn. Baseten’s /responses accepts reasoning.effort from minimal to high, a narrower range than chat. Bifrost sends none as minimal, and xhigh or max as high. Dedicated deployments have no /responses route. For an alias with deployment_url, Bifrost converts the request to chat completions and converts the reply back.

3. Embeddings

Embeddings run on dedicated BEI deployments, which serve the OpenAI embeddings shape.
my-embedder must be an alias with a deployment_url. A request for a model without one, including any Model APIs model, fails before reaching Baseten.

Rerank

Cross-encoder rerankers deployed on BEI with the /rerank route, such as the ModernBERT reranker in Baseten’s BEI examples, serve it under the deployment’s /sync base. Set the alias deployment_url to the deployment’s /sync/v1 URL, as for embeddings, and Bifrost calls /sync/rerank beside it.
Results come back highest relevance_score first, with each document’s index and id. BEI fields such as raw_scores, truncate and truncation_direction pass through as extra params with x-bf-passthrough-extra-params: true. Baseten’s Qwen3 Reranker and zerank recipes are deployed with /predict as their route and take model-specific prompts in inputs. Bifrost doesn’t build those prompts, so use a /rerank cross-encoder for /v1/rerank.

4. Transcriptions

Dedicated speech-recognition deployments (for example Qwen3 ASR) serve OpenAI’s multipart /audio/transcriptions, so /v1/audio/transcriptions works against an alias with deployment_url. Streaming transcription isn’t supported, because Baseten’s real-time transcription uses its own WebSocket protocol.

5. Image Generation

Baseten’s image recipes, such as FLUX.2, Krea 2 Turbo and Qwen-Image, run on the SGLang diffusion stack and serve OpenAI’s /images/generations, so /v1/images/generations works against an alias with deployment_url.
Images come back as base64 in data[].b64_json. The deployments can’t return image URLs, so when a request sets no response_format, Bifrost sends b64_json instead of leaving OpenAI’s url default in place. Streaming, edits and variations aren’t supported, because the deployments don’t serve those routes. A request for a model without a deployment_url, including any Model APIs model, fails before reaching Baseten.

Using the /predict URL from the dashboard

The Baseten dashboard shows each deployment’s /predict URL, for example https://model-def456.api.baseten.co/environments/production/predict. You can use it as deployment_url for image generation. Bifrost posts the same OpenAI image body to that URL as given, without appending /images/generations:
  • FLUX.2, Krea 2 Turbo and Qwen-Image map /predict to their /v1/images/generations route, so they behave as with the /sync/v1 URL.
  • FLUX.1 dev and schnell are packaged as a Truss that only serves /predict. It reads prompt, seed and num_inference_steps from the body, ignores the other fields, and returns one base64 image. size has no effect, so pass width and height as extra params with x-bf-passthrough-extra-params: true for anything other than 1024x1024.
Bifrost reads both reply shapes, the Truss’s {"data": "<base64>"} and OpenAI’s data[].b64_json, into data[].b64_json. A reply with no image fails rather than returning an empty list. A custom Truss with a different request or reply shape isn’t supported. A /predict URL only works for image generation. Chat, embeddings and transcription against that alias fail before any call, with an error that asks for the deployment’s /sync/v1 URL.

6. Decisions

Baseten’s decision models answer /v1/decisions through Model APIs’ System One endpoint, /systemone. They classify, route and score an input with typed answers instead of generated text.
How Bifrost maps the request:
  • predicate, choice and score questions become System One’s noul, choice and score. box questions aren’t supported and come back as Baseten’s 400. A request can have at most 255 choices per question, 2 to 10 score levels, and 512 options in total, where each predicate counts as two.
  • Text and structured input are sent as the state unchanged. A message input is sent as {"messages": [...]}, because Baseten reads a top-level array as data rather than a conversation. Only user and assistant roles are accepted, and a message without a role is sent as user. Any other role fails before reaching Baseten.
  • Images go inline as input_image parts on user messages, as a base64 PNG, JPEG or WebP data URL or an HTTPS URL on Amazon S3, Google Cloud Storage or Azure Blob Storage, up to 16 per request. An image detail other than auto is dropped, since Baseten accepts only auto. Images on assistant messages, more than 16 images, images sent by reference (with an id), and tool calls or outputs in the input fail before reaching Baseten. Bifrost passes images to every decision model, but Baseten trained elephant-1 and mammoth-1 on text only, so they may not read an image reliably.
  • Baseten’s service_tier (flex or priority) and max_len are sent when the request carries them as extra params with x-bf-passthrough-extra-params: true.
Bifrost sends Decisions to Model APIs and ignores an alias deployment_url, even though Baseten also offers decision models as dedicated deployments. With the compat plugin’s convert_decision_to_responses on, decision-models/ models still go to Baseten’s endpoint, and any other Baseten model is emulated through the Responses API.

7. List Models

Bifrost lists models from Model APIs’ /models for each key, plus the aliases configured on the key. Dedicated deployments appear through their aliases.

8. Errors and retries

Baseten returns 529 when a model is overloaded. Bifrost treats it as transient, so it is retried with the provider’s backoff settings when network_config.max_retries is above 0. max_retries defaults to 0, so set it before relying on retries; see Retries and fallbacks. Other errors keep their status code and message.

Baseten through Hugging Face

Hugging Face’s inference router also reaches Baseten-hosted models, as huggingface/baseten/<model>. That path uses your Hugging Face token and Hugging Face’s model IDs. The baseten provider calls Baseten directly with a Baseten API key, and it is the only way to reach your dedicated deployments. If you set up Baseten earlier as a custom provider named baseten, that name now belongs to this provider. The dashboard offers to rename the custom one so both can exist side by side.