Overview
Baseten serves open models two ways, and the Bifrostbaseten provider covers both:
- Model APIs. A shared, OpenAI-compatible endpoint at
https://inference.baseten.co/v1for models such aszai-org/GLM-5.3,deepseek-ai/DeepSeek-V4-Pro-0813andopenai/gpt-oss-120b. Bifrost uses it by default. - Dedicated deployments. A model you deployed yourself, with its own URL such as
https://model-<id>.api.baseten.co/environments/production/sync/v1. You point a model alias at it withdeployment_url.
Supported operations
Model APIs has no embeddings, rerank, audio or image routes, so those operations need a dedicated deployment. A request without one fails with an error that names the missing
deployment_url. Baseten’s TTS and classification deployments, and rerankers served on /predict, use their own request shapes, which Bifrost doesn’t translate. FLUX.1 image Trusses are the exception: see image generation. Baseten has no video generation models.
Setup and configuration
- Web UI
- config.json
- API
- Go SDK
- Navigate to Models > Model Providers. If Baseten isn’t under Configured Providers, click Add New Provider and select Baseten.
- Click Add Key or edit an existing key.
- Set a name for your key.
- Paste your Baseten API key directly or use an environment variable (for example,
env.BASETEN_API_KEY). - Save the provider configuration.
network_config.base_url changes the Model APIs host. Leave it empty to use https://inference.baseten.co/v1.
Dedicated deployments
Add an alias to the key and set itsdeployment_url to the deployment’s OpenAI-compatible base URL. Find it on the deployment’s page in the Baseten dashboard: it ends in /sync/v1. Requests for that alias go to the deployment instead of Model APIs, using the same API key.
The alias model_id is what Bifrost sends as model:
- Engine-Builder (TensorRT-LLM) and BEI deployments ignore
model, so any value works. Baseten’s examples usenot-required. - vLLM and custom servers need the name the server was started with (
--served-model-name). Any other value returns a 404.
model_name to the model the deployment runs, so pricing and logs show a real model name instead of not-required.
- Web UI
- config.json
- Edit the Baseten key and open Deployments.
- Add a deployment. The name is what clients send, for example
my-glm, and the model ID is the served model name ornot-required. - Expand the row and fill Deployment URL with the
/sync/v1URL, or an env reference such asenv.BASETEN_GLM_DEPLOYMENT_URL. - Save the key.
https.deployment_url must be an absolute https URL. Bifrost checks env and vault references when they resolve. If one resolves to an empty value, the request fails rather than falling back to Model APIs. deployment_url is a host Bifrost sends the key to, so it gets the same endpoint-change protection as Azure’s per-alias endpoint.
To target a named environment or a specific deployment instead of production, use that deployment’s own /sync/v1 URL from the dashboard.
1. Chat Completions
Request parameters
Baseten accepts the usual OpenAI chat parameters, includingtools, tool_choice, parallel_tool_calls, response_format (json_object and json_schema), logprobs, top_logprobs, seed, stop, temperature, top_p and the penalties. For the full parameter reference, see OpenAI Chat Completions.
Baseten’s chat schema rejects fields it doesn’t define, so Bifrost adjusts the request before sending it:
n and best_of only accept 1. Larger values come back as Baseten’s own 400.
Reasoning
reasoning_effort passes through with every level Baseten accepts: none, minimal, low, medium, high, xhigh and max. Models without reasoning ignore it. Baseten returns the thinking as reasoning_content, which Bifrost exposes as the message’s reasoning, and keeps it on assistant turns you send back. Reflection’s Beam returns a 400 when earlier assistant turns lack it.
GLM 5.2 and Nemotron Ultra turn thinking on through the chat template instead of reasoning_effort. Send it as an extra param:
Extra parameters
Extra parameters pass through unchanged, so Baseten-only sampling fields such astop_k, min_p, repetition_penalty, min_tokens and chat_template_args reach the API. Baseten’s documented name is chat_template_args, not chat_template_kwargs.
Streaming
Baseten sends usage on every streamed chunk by default. Bifrost reports it once, on the final chunk, with the cumulative totals.Prompt caching
Baseten caches prompt prefixes automatically. Sending the samex-session-affinity value on related requests improves hit rates. A client forwards it as x-bf-eh-x-session-affinity, or you can set it for every request in network_config.extra_headers.
2. Responses API
On Model APIs, Responses requests go to Baseten’s native/responses, streamed or not. It is stateless: previous_response_id, conversation, background and prompt return 501, so send the full input on every turn.
Baseten’s /responses accepts reasoning.effort from minimal to high, a narrower range than chat. Bifrost sends none as minimal, and xhigh or max as high.
Dedicated deployments have no /responses route. For an alias with deployment_url, Bifrost converts the request to chat completions and converts the reply back.
3. Embeddings
Embeddings run on dedicated BEI deployments, which serve the OpenAI embeddings shape.my-embedder must be an alias with a deployment_url. A request for a model without one, including any Model APIs model, fails before reaching Baseten.
Rerank
Cross-encoder rerankers deployed on BEI with the/rerank route, such as the ModernBERT reranker in Baseten’s BEI examples, serve it under the deployment’s /sync base. Set the alias deployment_url to the deployment’s /sync/v1 URL, as for embeddings, and Bifrost calls /sync/rerank beside it.
Results come back highest
relevance_score first, with each document’s index and id. BEI fields such as raw_scores, truncate and truncation_direction pass through as extra params with x-bf-passthrough-extra-params: true.
Baseten’s Qwen3 Reranker and zerank recipes are deployed with /predict as their route and take model-specific prompts in inputs. Bifrost doesn’t build those prompts, so use a /rerank cross-encoder for /v1/rerank.
4. Transcriptions
Dedicated speech-recognition deployments (for example Qwen3 ASR) serve OpenAI’s multipart/audio/transcriptions, so /v1/audio/transcriptions works against an alias with deployment_url. Streaming transcription isn’t supported, because Baseten’s real-time transcription uses its own WebSocket protocol.
5. Image Generation
Baseten’s image recipes, such as FLUX.2, Krea 2 Turbo and Qwen-Image, run on the SGLang diffusion stack and serve OpenAI’s/images/generations, so /v1/images/generations works against an alias with deployment_url.
data[].b64_json. The deployments can’t return image URLs, so when a request sets no response_format, Bifrost sends b64_json instead of leaving OpenAI’s url default in place. Streaming, edits and variations aren’t supported, because the deployments don’t serve those routes. A request for a model without a deployment_url, including any Model APIs model, fails before reaching Baseten.
Using the /predict URL from the dashboard
The Baseten dashboard shows each deployment’s/predict URL, for example https://model-def456.api.baseten.co/environments/production/predict. You can use it as deployment_url for image generation. Bifrost posts the same OpenAI image body to that URL as given, without appending /images/generations:
- FLUX.2, Krea 2 Turbo and Qwen-Image map
/predictto their/v1/images/generationsroute, so they behave as with the/sync/v1URL. - FLUX.1 dev and schnell are packaged as a Truss that only serves
/predict. It readsprompt,seedandnum_inference_stepsfrom the body, ignores the other fields, and returns one base64 image.sizehas no effect, so passwidthandheightas extra params withx-bf-passthrough-extra-params: truefor anything other than 1024x1024.
{"data": "<base64>"} and OpenAI’s data[].b64_json, into data[].b64_json. A reply with no image fails rather than returning an empty list. A custom Truss with a different request or reply shape isn’t supported.
A /predict URL only works for image generation. Chat, embeddings and transcription against that alias fail before any call, with an error that asks for the deployment’s /sync/v1 URL.
6. Decisions
Baseten’s decision models answer/v1/decisions through Model APIs’ System One endpoint, /systemone. They classify, route and score an input with typed answers instead of generated text.
predicate,choiceandscorequestions become System One’snoul,choiceandscore.boxquestions aren’t supported and come back as Baseten’s 400. A request can have at most 255 choices per question, 2 to 10 score levels, and 512 options in total, where each predicate counts as two.- Text and structured input are sent as the
stateunchanged. A message input is sent as{"messages": [...]}, because Baseten reads a top-level array as data rather than a conversation. Onlyuserandassistantroles are accepted, and a message without a role is sent asuser. Any other role fails before reaching Baseten. - Images go inline as
input_imageparts on user messages, as a base64 PNG, JPEG or WebP data URL or an HTTPS URL on Amazon S3, Google Cloud Storage or Azure Blob Storage, up to 16 per request. An imagedetailother thanautois dropped, since Baseten accepts onlyauto. Images on assistant messages, more than 16 images, images sent by reference (with anid), and tool calls or outputs in the input fail before reaching Baseten. Bifrost passes images to every decision model, but Baseten trained elephant-1 and mammoth-1 on text only, so they may not read an image reliably. - Baseten’s
service_tier(flexorpriority) andmax_lenare sent when the request carries them as extra params withx-bf-passthrough-extra-params: true.
deployment_url, even though Baseten also offers decision models as dedicated deployments. With the compat plugin’s convert_decision_to_responses on, decision-models/ models still go to Baseten’s endpoint, and any other Baseten model is emulated through the Responses API.
7. List Models
Bifrost lists models from Model APIs’/models for each key, plus the aliases configured on the key. Dedicated deployments appear through their aliases.
8. Errors and retries
Baseten returns 529 when a model is overloaded. Bifrost treats it as transient, so it is retried with the provider’s backoff settings whennetwork_config.max_retries is above 0. max_retries defaults to 0, so set it before relying on retries; see Retries and fallbacks. Other errors keep their status code and message.
Baseten through Hugging Face
Hugging Face’s inference router also reaches Baseten-hosted models, ashuggingface/baseten/<model>. That path uses your Hugging Face token and Hugging Face’s model IDs. The baseten provider calls Baseten directly with a Baseten API key, and it is the only way to reach your dedicated deployments.
If you set up Baseten earlier as a custom provider named baseten, that name now belongs to this provider. The dashboard offers to rename the custom one so both can exist side by side.
