​

Overview

• OpenAI compatibility - Chat, text completions, embeddings, and streaming
• Self-hosted - Typically runs at http://localhost:8000 or your own server
• Optional authentication - API key often omitted for local instances
• Responses API - Supported via chat completion fallback

​

Supported Operations

​

Authentication

• API key: Optional. For local vLLM instances, the key is often left empty.
• When set, the key is sent as Authorization: Bearer <key>.

​

Configuration

• Base URL: Default is http://localhost:8000. Override via provider network_config.base_url.
• Model names: Depend on the models loaded in your vLLM instance (e.g. meta-llama/Llama-3.2-1B-Instruct, BAAI/bge-m3 for embeddings).

# Point to local or remote vLLM instance (default: http://localhost:8000)
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vllm/meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

# Gateway provider config: set base_url for remote vLLM
# "network_config": { "base_url": "http://vllm-endpoint:8000" }

config := &schemas.ProviderConfig{
    NetworkConfig: schemas.NetworkConfig{
        BaseURL: "http://localhost:8000",  // optional; default is http://localhost:8000
        DefaultRequestTimeoutInSeconds: 30,
    },
}
provider, _ := vllm.NewVLLMProvider(config, logger)

response, _ := provider.ChatCompletion(ctx, key, request)

​

Getting started

1. Run a vLLM server (Docker or pip). Example with Docker:
   Copy

   Ask AI

   docker run --gpus all -p 8000:8000 vllm/vllm-openai:latest --model meta-llama/Llama-3.2-1B-Instruct
   
2. Verify the server:
   Copy

   Ask AI

   curl http://localhost:8000/v1/models
   
3. Use Bifrost with model prefix vllm/<model_id> (e.g. vllm/meta-llama/Llama-3.2-1B-Instruct).

​

1. Chat Completions

​

2. Responses API

BifrostResponsesRequest
  → ToChatRequest()
  → ChatCompletion
  → ToBifrostResponsesResponse()

​

3. Text Completions

​

4. Embeddings

​

5. List Models

​

Caveats