HTTP Server] Router[Request Router
Goroutine] PluginMgr[Plugin Manager
Goroutine] end subgraph "Provider Worker Pools" subgraph "OpenAI Pool" OAI1[Worker 1
Goroutine] OAI2[Worker 2
Goroutine] OAIN[Worker N
Goroutine] end subgraph "Anthropic Pool" ANT1[Worker 1
Goroutine] ANT2[Worker 2
Goroutine] ANTN[Worker N
Goroutine] end subgraph "Bedrock Pool" BED1[Worker 1
Goroutine] BED2[Worker 2
Goroutine] BEDN[Worker N
Goroutine] end end subgraph "Memory Pools" ChannelPool[Channel Pool
sync.Pool] MessagePool[Message Pool
sync.Pool] ResponsePool[Response Pool
sync.Pool] end Main --> Router Router --> PluginMgr PluginMgr --> OAI1 PluginMgr --> ANT1 PluginMgr --> BED1 OAI1 --> ChannelPool ANT1 --> MessagePool BED1 --> ResponsePool ``` *** ## Worker Pool Architecture ### **Provider-Isolated Worker Pools** ```mermaid theme={null} stateDiagram-v2 [*] --> PoolInit: Worker Pool Creation PoolInit --> WorkerSpawn: Spawn Worker Goroutines WorkerSpawn --> Listening: Workers Listen on Channels Listening --> Processing: Job Received Processing --> API_Call: Provider API Request API_Call --> Response: Process Response Response --> Listening: Job Complete Listening --> Shutdown: Graceful Shutdown Processing --> Shutdown: Complete Current Job Shutdown --> [*]: Pool Destroyed ``` **Worker Pool Architecture:** The worker pool system maintains a sophisticated balance between resource efficiency and performance isolation: **Key Components:** * **Worker Pool Management** - Pre-spawned workers reduce startup latency * **Job Queue System** - Buffered channels provide smooth load balancing * **Resource Pools** - HTTP clients and API keys are pooled for efficiency * **Health Monitoring** - Circuit breakers detect and isolate failing providers * **Graceful Shutdown** - Workers complete current jobs before terminating **Startup Process:** 1. **Worker Pre-spawning** - Workers are created during pool initialization 2. **Channel Setup** - Job queues and worker channels are established 3. **Resource Allocation** - HTTP clients and API keys are distributed 4. **Health Checks** - Initial connectivity tests verify provider availability 5. **Ready State** - Pool becomes available for request processing **Job Dispatch Logic:** * **Round-Robin Assignment** - Jobs are distributed evenly across available workers * **Load Balancing** - Worker availability determines job assignment * **Overflow Handling** - Excess jobs are queued or dropped based on configuration ### **Worker Lifecycle Management** ```mermaid theme={null} sequenceDiagram participant Pool participant Worker participant HTTPClient participant Provider participant Metrics Pool->>Worker: Start() Worker->>Worker: Initialize HTTP Client Worker->>Pool: Ready Signal loop Job Processing Pool->>Worker: Job Assignment Worker->>HTTPClient: Prepare Request HTTPClient->>Provider: API Call Provider-->>HTTPClient: Response HTTPClient-->>Worker: Parsed Response Worker->>Metrics: Record Performance Worker->>Pool: Job Complete end Pool->>Worker: Shutdown Signal Worker->>Worker: Complete Current Job Worker-->>Pool: Shutdown Confirmed ``` *** ## Channel-Based Communication ### **Channel Architecture** ```mermaid theme={null} graph TB subgraph "Channel Types" JobQueue[Job Queue
Buffered Channel] WorkerPool[Worker Pool
Buffered Channel] ResultChan[Result Channel
Buffered Channel] QuitChan[Quit Channel
Unbuffered] end subgraph "Flow Control" BackPressure[Backpressure
Buffer Limits] Timeout[Timeout
Context Cancellation] Graceful[Graceful Shutdown
Channel Closing] end JobQueue --> BackPressure WorkerPool --> Timeout ResultChan --> Graceful ``` **Channel Configuration Principles:** Bifrost's channel system balances throughput and memory usage through careful buffer sizing: **Job Queuing Configuration:** * **Job Queue Buffer** - Sized based on expected burst traffic (100-1000 jobs) * **Worker Pool Size** - Matches provider concurrency limits (10-100 workers) * **Result Buffer** - Accommodates response processing delays (50-500 responses) **Flow Control Parameters:** * **Queue Wait Limits** - Maximum time jobs wait before timeout (1-10 seconds) * **Processing Timeouts** - Per-job execution limits (30-300 seconds) * **Shutdown Timeouts** - Graceful termination periods (5-30 seconds) **Backpressure Policies:** * **Drop Policy** - Discard excess jobs when queues are full * **Block Policy** - Wait for queue space with timeout * **Error Policy** - Immediately return error for full queues **Channel Type Selection:** * **Buffered Channels** - Used for async job processing and result handling * **Unbuffered Channels** - Used for synchronization signals (quit, done) * **Context Cancellation** - Used for timeout and cancellation propagation ### **Backpressure and Flow Control** ```mermaid theme={null} flowchart TD Request[Incoming Request] --> QueueCheck{Queue Full?} QueueCheck -->|No| Queue[Add to Queue] QueueCheck -->|Yes| Policy{Drop Policy?} Policy -->|Drop| Drop[Drop Request
Return Error] Policy -->|Block| Block[Block Until Space
With Timeout] Policy -->|Error| Error[Return Queue Full Error] Queue --> Worker[Assign to Worker] Block --> TimeoutCheck{Timeout?} TimeoutCheck -->|Yes| Error TimeoutCheck -->|No| Queue Worker --> Processing[Process Request] Processing --> Complete[Complete] Drop --> Client[Client Response] Error --> Client Complete --> Client ``` **Backpressure Implementation Strategy:** The backpressure system protects Bifrost from being overwhelmed while maintaining service availability: **Non-Blocking Job Submission:** * **Immediate Queue Check** - Jobs are submitted without blocking on queue space * **Success Path** - Available queue space allows immediate job acceptance * **Overflow Detection** - Full queues trigger backpressure policies * **Metrics Collection** - All queue operations are tracked for monitoring **Backpressure Policy Execution:** * **Drop Policy** - Immediately rejects excess jobs with meaningful error messages * **Block Policy** - Waits for queue space with configurable timeout limits * **Error Policy** - Returns queue full errors for immediate client feedback * **Metrics Tracking** - Dropped, blocked, and successful submissions are measured **Timeout Management:** * **Context-Based Timeouts** - All blocking operations respect timeout boundaries * **Graceful Degradation** - Timeouts result in controlled error responses * **Resource Protection** - Prevents goroutine leaks from infinite waits ```go theme={null} case pool.jobQueue <- job: pool.metrics.IncQueuedJobs() return nil case <-ctx.Done(): pool.metrics.IncTimeoutJobs() return errors.New("queue full, timeout waiting") } case "error": pool.metrics.IncRejectedJobs() return errors.New("queue full, job rejected") default: return errors.New("unknown queue policy") } } } ``` *** ## Memory Pool Concurrency ### **Thread-Safe Object Pools** ```mermaid theme={null} graph TD subgraph "sync.Pool Lifecycle" direction LR GetObject[Get Object
sync.Pool.Get] PoolCheck{Is Pool Empty?} NewObject[New Object
Factory Function] UseObject[Use Object
Application Logic] ResetObject[Reset Object
Clear State] ReturnObject[Return Object
sync.Pool.Put] GetObject --> PoolCheck PoolCheck -- Yes --> NewObject PoolCheck -- No --> UseObject NewObject --> UseObject UseObject --> ResetObject ResetObject --> ReturnObject ReturnObject --> GetObject end subgraph "GC Interaction" direction TB GCRun[GC Runs] PoolCleanup[Pool Cleanup
Removes idle objects] GCRun --> PoolCleanup end ``` **Thread-Safe Pool Architecture:** Bifrost's memory pool system ensures thread-safe object reuse across multiple goroutines: **Pool Structure Design:** * **Multiple Pool Types** - Separate pools for channels, messages, responses, and buffers * **Factory Functions** - Dynamic object creation when pools are empty * **Statistics Tracking** - Comprehensive metrics for pool performance monitoring * **Thread Safety** - Synchronized access using Go's sync.Pool and read-write mutexes **Object Lifecycle Management:** * **Pool Initialization** - Factory functions define object creation patterns * **Unique Identification** - Each pooled object gets a unique ID for tracking * **Timestamp Tracking** - Creation, acquisition, and return times are recorded * **Reusability Flags** - Objects can be marked as non-reusable for single-use scenarios **Acquisition Strategy:** * **Request Tracking** - All pool requests are counted for monitoring * **Hit/Miss Tracking** - Pool effectiveness is measured through hit ratios * **Fallback Creation** - New objects are created when pools are empty * **Performance Metrics** - Acquisition times and patterns are monitored **Return and Reset Process:** * **State Validation** - Only reusable objects are returned to pools * **Object Reset** - All object state is cleared before returning to pool * **Return Tracking** - Return operations are counted and timed * **Pool Replenishment** - Returned objects become available for reuse ### **Pool Performance Monitoring** Comprehensive metrics provide insights into pool efficiency and system health: **Usage Statistics Collection:** * **Request Counting** - Track total pool requests by object type * **Creation Tracking** - Monitor new object allocations when pools are empty * **Hit/Miss Ratios** - Measure pool effectiveness through reuse rates * **Return Monitoring** - Track successful object returns to pools **Performance Metrics Analysis:** * **Acquisition Times** - Measure how long it takes to get objects from pools * **Reset Performance** - Track time spent cleaning objects for reuse * **Hit Ratio Calculation** - Determine percentage of requests served from pools * **Memory Efficiency** - Calculate memory savings from object reuse **Key Performance Indicators:** * **Channel Pool Hit Ratio** - Typically 85-95% in steady state * **Message Pool Efficiency** - Usually 80-90% reuse rate * **Response Pool Utilization** - Often 70-85% hit ratio * **Total Memory Savings** - Measured reduction in garbage collection pressure **Monitoring Integration:** * **Thread-Safe Access** - All metrics collection is synchronized * **Real-Time Updates** - Statistics are updated with each pool operation * **Export Capability** - Metrics are available in JSON format for monitoring systems * **Alerting Support** - Low hit ratios can trigger performance alerts *** ## Goroutine Management ### **Goroutine Lifecycle Patterns** ```mermaid theme={null} stateDiagram-v2 [*] --> Created: go routine() Created --> Running: Execute Function Running --> Waiting: Channel/Mutex Block Waiting --> Running: Unblocked Running --> Syscall: Network I/O Syscall --> Running: I/O Complete Running --> GCAssist: GC Triggered GCAssist --> Running: GC Complete Running --> Terminated: Function Exit Terminated --> [*]: Cleanup ``` **Goroutine Pool Management Strategy:** Bifrost's goroutine management ensures optimal resource usage while preventing goroutine leaks: **Pool Configuration Management:** * **Goroutine Limits** - Maximum concurrent goroutines prevent resource exhaustion * **Active Counting** - Atomic counters track currently running goroutines * **Idle Timeouts** - Unused goroutines are cleaned up after configured periods * **Resource Boundaries** - Hard limits prevent runaway goroutine creation **Lifecycle Orchestration:** * **Spawn Channels** - New goroutine creation is tracked through channels * **Completion Monitoring** - Finished goroutines signal completion for cleanup * **Shutdown Coordination** - Graceful shutdown ensures all goroutines complete properly * **Health Monitoring** - Continuous monitoring tracks goroutine health and performance **Worker Creation Process:** * **Limit Enforcement** - Creation fails when maximum goroutine count is reached * **Unique Identification** - Each goroutine gets a unique ID for tracking and debugging * **Lifecycle Tracking** - Start times and names enable performance analysis * **Atomic Operations** - Thread-safe counters prevent race conditions **Panic Recovery and Error Handling:** * **Panic Isolation** - Goroutine panics don't crash the entire system * **Error Logging** - Panic details are logged with goroutine context * **Metrics Updates** - Panic counts are tracked for monitoring and alerting * **Resource Cleanup** - Failed goroutines are properly cleaned up and counted **Health Monitoring System:** * **Periodic Health Checks** - Regular intervals check goroutine pool health * **Completion Tracking** - Finished goroutines are recorded for performance analysis * **Shutdown Handling** - Clean shutdown process ensures no goroutine leaks ### **Resource Leak Prevention** ```mermaid theme={null} flowchart TD GoroutineStart[Goroutine Start] --> ResourceCheck[Resource Allocation Check] ResourceCheck --> Timeout[Set Timeout Context] Timeout --> Work[Execute Work] Work --> Complete{Work Complete?} Complete -->|Yes| Cleanup[Cleanup Resources] Complete -->|No| TimeoutCheck{Timeout?} TimeoutCheck -->|Yes| ForceCleanup[Force Cleanup] TimeoutCheck -->|No| Work Cleanup --> Return[Return Resources to Pool] ForceCleanup --> Return Return --> End[Goroutine End] ``` **Resource Leak Prevention:** ```go theme={null} func (worker *Worker) ExecuteWithCleanup(job *Job) { // Set timeout context ctx, cancel := context.WithTimeout( context.Background(), worker.config.ProcessTimeout, ) defer cancel() // Acquire resources with timeout resources, err := worker.acquireResources(ctx) if err != nil { job.resultChan <- &Result{Error: err} return } // Ensure cleanup happens defer func() { // Always return resources worker.returnResources(resources) // Handle panics if r := recover(); r != nil { worker.metrics.IncPanics() job.resultChan <- &Result{ Error: fmt.Errorf("worker panic: %v", r), } } }() // Execute job with context result := worker.processJob(ctx, job, resources) // Return result select { case job.resultChan <- result: // Success case <-ctx.Done(): // Timeout - result channel might be closed worker.metrics.IncTimeouts() } } ``` *** ## Concurrency Optimization Strategies ### **Load-Based Worker Scaling** (Planned) ```mermaid theme={null} graph TB subgraph "Load Monitoring" QueueDepth[Queue Depth
Monitoring] ResponseTime[Response Time
Tracking] WorkerUtil[Worker Utilization
Metrics] end subgraph "Scaling Decisions" ScaleUp{Scale Up?
Load > 80%} ScaleDown{Scale Down?
Load < 30%} Maintain[Maintain
Current Size] end subgraph "Actions" AddWorkers[Spawn Additional
Workers] RemoveWorkers[Graceful Worker
Shutdown] NoAction[No Action
Monitor Continue] end QueueDepth --> ScaleUp ResponseTime --> ScaleUp WorkerUtil --> ScaleDown ScaleUp -->|Yes| AddWorkers ScaleUp -->|No| ScaleDown ScaleDown -->|Yes| RemoveWorkers ScaleDown -->|No| Maintain Maintain --> NoAction ``` **Adaptive Scaling Implementation:** ```go theme={null} type AdaptiveScaler struct { pool *ProviderWorkerPool config ScalingConfig metrics *ScalingMetrics lastScaleTime time.Time scalingMutex sync.Mutex } func (scaler *AdaptiveScaler) EvaluateScaling() { scaler.scalingMutex.Lock() defer scaler.scalingMutex.Unlock() // Prevent frequent scaling if time.Since(scaler.lastScaleTime) < scaler.config.MinScaleInterval { return } current := scaler.getCurrentMetrics() // Scale up conditions if current.QueueUtilization > scaler.config.ScaleUpThreshold || current.AvgResponseTime > scaler.config.MaxResponseTime { scaler.scaleUp(current) return } // Scale down conditions if current.QueueUtilization < scaler.config.ScaleDownThreshold && current.AvgResponseTime < scaler.config.TargetResponseTime { scaler.scaleDown(current) return } } func (scaler *AdaptiveScaler) scaleUp(metrics *CurrentMetrics) { currentWorkers := scaler.pool.GetWorkerCount() targetWorkers := int(float64(currentWorkers) * scaler.config.ScaleUpFactor) // Respect maximum limits if targetWorkers > scaler.config.MaxWorkers { targetWorkers = scaler.config.MaxWorkers } additionalWorkers := targetWorkers - currentWorkers if additionalWorkers > 0 { scaler.pool.AddWorkers(additionalWorkers) scaler.lastScaleTime = time.Now() scaler.metrics.RecordScaleUp(additionalWorkers) } } ``` ### **Provider-Specific Optimization** ```go theme={null} type ProviderOptimization struct { // Provider characteristics ProviderName string `json:"provider_name"` RateLimit int `json:"rate_limit"` // Requests per second AvgLatency time.Duration `json:"avg_latency"` // Average response time ErrorRate float64 `json:"error_rate"` // Historical error rate // Optimal configuration OptimalWorkers int `json:"optimal_workers"` OptimalBuffer int `json:"optimal_buffer"` TimeoutConfig time.Duration `json:"timeout_config"` RetryStrategy RetryConfig `json:"retry_strategy"` } func CalculateOptimalConcurrency(provider ProviderOptimization) ConcurrencyConfig { // Calculate based on rate limits and latency optimalWorkers := provider.RateLimit * int(provider.AvgLatency.Seconds()) // Adjust for error rate (more workers for higher error rate) errorAdjustment := 1.0 + provider.ErrorRate optimalWorkers = int(float64(optimalWorkers) * errorAdjustment) // Buffer should be 2-3x worker count for smooth operation optimalBuffer := optimalWorkers * 3 return ConcurrencyConfig{ Concurrency: optimalWorkers, BufferSize: optimalBuffer, Timeout: provider.AvgLatency * 2, // 2x avg latency for timeout } } ``` *** ## Concurrency Monitoring & Metrics ### **Key Concurrency Metrics** ```mermaid theme={null} graph TB subgraph "Worker Metrics" ActiveWorkers[Active Workers
Current Count] IdleWorkers[Idle Workers
Available Count] BusyWorkers[Busy Workers
Processing Count] end subgraph "Queue Metrics" QueueDepth[Queue Depth
Pending Jobs] QueueThroughput[Queue Throughput
Jobs/Second] QueueWaitTime[Queue Wait Time
Average Delay] end subgraph "Performance Metrics" GoroutineCount[Goroutine Count
Total Active] MemoryUsage[Memory Usage
Pool Utilization] GCPressure[GC Pressure
Collection Frequency] end subgraph "Health Metrics" ErrorRate[Error Rate
Failed Jobs %] PanicCount[Panic Count
Crashed Goroutines] DeadlockDetection[Deadlock Detection
Blocked Operations] end ``` **Metrics Collection Strategy:** Comprehensive concurrency monitoring provides operational insights and performance optimization data: **Worker Pool Monitoring:** * **Total Worker Tracking** - Monitor configured vs actual worker counts * **Active Worker Monitoring** - Track workers currently processing requests * **Idle Worker Analysis** - Identify unused capacity and optimization opportunities * **Queue Depth Monitoring** - Track pending job backlog and processing delays **Performance Data Collection:** * **Throughput Metrics** - Measure jobs processed per second across all pools * **Wait Time Analysis** - Track how long jobs wait in queues before processing * **Memory Pool Performance** - Monitor hit/miss ratios for memory pool effectiveness * **Goroutine Count Tracking** - Ensure goroutine counts remain within healthy limits **Health and Reliability Metrics:** * **Panic Recovery Tracking** - Count and analyze worker panic occurrences * **Timeout Monitoring** - Track jobs that exceed processing time limits * **Circuit Breaker Events** - Monitor provider isolation events and recoveries * **Error Rate Analysis** - Track failure patterns for capacity planning **Real-Time Updates:** * **Live Metric Updates** - Worker metrics are updated continuously during operation * **Processing Event Recording** - Each job completion updates relevant metrics * **Performance Correlation** - Queue times and processing times are correlated for analysis * **Success/Failure Tracking** - All job outcomes are recorded for reliability analysis *** ## Deadlock Prevention & Detection ### **Deadlock Prevention Strategies** ```mermaid theme={null} flowchart TD Strategy1[Lock Ordering
Consistent Acquisition] Strategy2[Timeout-Based Locks
Context Cancellation] Strategy3[Channel Select
Non-blocking Operations] Strategy4[Resource Hierarchy
Layered Locking] Prevention[Deadlock Prevention
Design Patterns] Prevention --> Strategy1 Prevention --> Strategy2 Prevention --> Strategy3 Prevention --> Strategy4 Strategy1 --> Success[No Deadlocks
Guaranteed Order] Strategy2 --> Success Strategy3 --> Success Strategy4 --> Success ``` **Deadlock Prevention Implementation Strategy:** Bifrost employs multiple complementary strategies to prevent deadlocks in concurrent operations: **Lock Ordering Management:** * **Consistent Acquisition Order** - All locks are acquired in a predetermined order * **Global Lock Registry** - Centralized registry maintains lock ordering relationships * **Order Enforcement** - Lock acquisition automatically sorts by predetermined order * **Dependency Tracking** - Lock dependencies are mapped to prevent circular waits **Timeout-Based Protection:** * **Default Timeouts** - All lock acquisitions have reasonable timeout limits * **Context Cancellation** - Operations respect context cancellation for cleanup * **Maximum Timeout Limits** - Upper bounds prevent indefinite blocking * **Graceful Timeout Handling** - Timeout errors provide meaningful context **Multi-Lock Acquisition Process:** * **Ordered Sorting** - Multiple locks are sorted before acquisition attempts * **Progressive Acquisition** - Locks are acquired one by one in sorted order * **Failure Recovery** - Failed acquisitions trigger automatic cleanup of held locks * **Resource Tracking** - All acquired locks are tracked for proper release **Lock Acquisition Safety:** * **Non-Blocking Detection** - Channel-based lock attempts prevent indefinite blocking * **Timeout Enforcement** - All lock attempts respect configured timeout limits * **Error Propagation** - Lock failures are properly propagated with context * **Cleanup Guarantees** - Failed operations always clean up partially acquired resources **Deadlock Detection and Recovery:** * **Active Monitoring** - Continuous monitoring for potential deadlock conditions * **Automatic Recovery** - Detected deadlocks trigger automatic resolution procedures * **Resource Release** - Deadlock resolution involves strategic resource release * **Prevention Learning** - Deadlock patterns inform prevention strategy improvements *** ## Related Architecture Documentation * **[Request Flow](./request-flow)** - How concurrency fits in request processing * **[Benchmarks](../../benchmarking/getting-started)** - Concurrency performance characteristics * **[Plugin System](./plugins)** - Plugin concurrency considerations * **[MCP System](./mcp)** - MCP concurrency and worker integration ## Usage Documentation * **[Provider Configuration](../../quickstart/gateway/provider-configuration)** - Configure concurrency settings per provider * **[Performance Analysis](../../benchmarking/getting-started)** - Memory pool configuration and optimization * **[Performance Monitoring](../../features/telemetry)** - Monitor concurrency metrics and health * **[Go SDK Usage](../../quickstart/go-sdk/setting-up)** - Use Bifrost concurrency in Go applications * **[Gateway Setup](../../quickstart/gateway/setting-up)** - Deploy Bifrost with optimal concurrency settings *** **🎯 Next Step:** Understand how plugins integrate with the concurrency model in **[Plugin System](./plugins)**. ``` ``` # Model Context Protocol (MCP) Source: https://docs.getbifrost.ai/architecture/core/mcp Deep dive into Bifrost's Model Context Protocol (MCP) integration - how external tool discovery, execution, and integration work internally. ## MCP Architecture Overview ### **What is MCP in Bifrost?** The Model Context Protocol (MCP) system in Bifrost enables AI models to seamlessly discover and execute external tools, transforming static chat models into dynamic, action-capable agents. This architecture bridges the gap between AI reasoning and real-world tool execution. **Core MCP Principles:** * **Dynamic Discovery** - Tools are discovered at runtime, not hardcoded * **Client-Side Execution** - Bifrost controls all tool execution for security * **Multi-Protocol Support** - STDIO, HTTP, and SSE connection types * **Request-Level Filtering** - Granular control over tool availability * **Async Execution** - Non-blocking tool invocation and response handling ### **MCP System Components** ```mermaid theme={null} graph TB subgraph "MCP Management Layer" MCPMgr[MCP Manager
Central Controller] ClientRegistry[Client Registry
Connection Management] ToolDiscovery[Tool Discovery
Runtime Registration] end subgraph "MCP Execution Layer" ToolFilter[Tool Filter
Access Control] ToolExecutor[Tool Executor
Invocation Engine] ResultProcessor[Result Processor
Response Handling] end subgraph "Connection Types" STDIOConn[STDIO Connections
Command-line Tools] HTTPConn[HTTP Connections
Web Services] SSEConn[SSE Connections
Real-time Streams] end subgraph "External MCP Servers" FileSystem[Filesystem Tools
File Operations] WebSearch[Web Search
Information Retrieval] Database[Database Tools
Data Access] Custom[Custom Tools
Business Logic] end MCPMgr --> ClientRegistry ClientRegistry --> ToolDiscovery ToolDiscovery --> ToolFilter ToolFilter --> ToolExecutor ToolExecutor --> ResultProcessor ClientRegistry --> STDIOConn ClientRegistry --> HTTPConn ClientRegistry --> SSEConn STDIOConn --> FileSystem HTTPConn --> WebSearch HTTPConn --> Database STDIOConn --> Custom ``` *** ## MCP Connection Architecture ### **Multi-Protocol Connection System** Bifrost supports four MCP connection types, each optimized for different tool deployment patterns: ```mermaid theme={null} graph TB subgraph "InProcess Connections" InProcess[In-Memory Tools
Same Process] InProcessEx[Examples:
• Embedded tools
• High-perf operations
• Testing tools] end subgraph "STDIO Connections" STDIO[Command Line Tools
Local Execution] STDIOEx[Examples:
• Filesystem tools
• Local scripts
• CLI utilities] end subgraph "HTTP Connections" HTTP[Web Service Tools
Remote APIs] HTTPEx[Examples:
• Web search APIs
• Database services
• External integrations] end subgraph "SSE Connections" SSE[Real-time Tools
Streaming Data] SSEEx[Examples:
• Live data feeds
• Real-time monitoring
• Event streams] end subgraph "Connection Characteristics" Latency[Latency:
InProcess < STDIO < HTTP < SSE] Security[Security:
InProcess/Local > HTTP > SSE] Scalability[Scalability:
HTTP > SSE > STDIO > InProcess] Complexity[Complexity:
InProcess < STDIO < HTTP < SSE] end InProcess --> Latency STDIO --> Latency HTTP --> Security SSE --> Scalability HTTP --> Complexity ``` ### **Connection Type Details** **InProcess Connections (In-Memory Tools):** * **Use Case:** Embedded tools, high-performance operations, testing * **Performance:** Lowest possible latency (\~0.1ms) with no IPC overhead * **Security:** Highest security as tools run in the same process * **Limitations:** Go package only, cannot be configured via JSON **STDIO Connections (Local Tools):** * **Use Case:** Command-line tools, local scripts, filesystem operations * **Performance:** Low latency (\~1-10ms) due to local execution * **Security:** High security with full local control * **Limitations:** Single-server deployment, resource sharing **HTTP Connections (Remote Services):** * **Use Case:** Web APIs, microservices, cloud functions * **Performance:** Network-dependent latency (\~10-500ms) * **Security:** Configurable with authentication and encryption * **Advantages:** Scalable, multi-server deployment, service isolation **SSE Connections (Streaming Tools):** * **Use Case:** Real-time data feeds, live monitoring, event streams * **Performance:** Variable latency depending on stream frequency * **Security:** Similar to HTTP with streaming capabilities * **Benefits:** Real-time updates, persistent connections, event-driven > **MCP Configuration:** [MCP Setup Guide →](../../mcp/overview) *** ## Tool Discovery & Registration ### **Dynamic Tool Discovery Process** The MCP system discovers tools at runtime rather than requiring static configuration, enabling flexible and adaptive tool availability: ```mermaid theme={null} sequenceDiagram participant Bifrost participant MCPManager participant MCPServer participant ToolRegistry participant AIModel Note over Bifrost: System Startup Bifrost->>MCPManager: Initialize MCP System MCPManager->>MCPServer: Establish Connection MCPServer-->>MCPManager: Connection Ready MCPManager->>MCPServer: List Available Tools MCPServer-->>MCPManager: Tool Definitions MCPManager->>ToolRegistry: Register Tools Note over Bifrost: Runtime Request Processing AIModel->>MCPManager: Request Available Tools MCPManager->>ToolRegistry: Query Tools ToolRegistry-->>MCPManager: Filtered Tool List MCPManager-->>AIModel: Available Tools AIModel->>MCPManager: Execute Tool Call MCPManager->>MCPServer: Tool Invocation MCPServer->>MCPServer: Execute Tool Logic MCPServer-->>MCPManager: Tool Result MCPManager-->>AIModel: Enhanced Response ``` ### **Tool Registry Management** **Registration Process:** 1. **Connection Establishment** - MCP client connects to configured servers 2. **Capability Exchange** - Server announces available tools and schemas 3. **Tool Validation** - Bifrost validates tool definitions and security 4. **Registry Update** - Tools are registered in the internal tool registry 5. **Availability Notification** - Tools become available for AI model use **Registry Features:** * **Dynamic Updates** - Tools can be added/removed during runtime * **Version Management** - Support for tool versioning and compatibility * **Access Control** - Request-level tool filtering and permissions * **Health Monitoring** - Continuous tool availability checking **Tool Metadata Structure:** * **Name & Description** - Human-readable tool identification * **Parameters Schema** - JSON schema for tool input validation * **Return Schema** - Expected response format definition * **Capabilities** - Tool feature flags and limitations * **Authentication** - Required credentials and permissions *** ## Tool Filtering & Access Control ### **Multi-Level Filtering System** Bifrost provides granular control over tool availability through a sophisticated filtering system: ```mermaid theme={null} flowchart TD Request[Incoming Request] --> GlobalFilter{Global MCP Filter} GlobalFilter -->|Enabled| ClientFilter[MCP Client Filtering] GlobalFilter -->|Disabled| NoMCP[No MCP Tools] ClientFilter --> IncludeClients{Include Clients?} IncludeClients -->|Yes| IncludeList[Include Specified
MCP Clients] IncludeClients -->|No| AllClients[All MCP Clients] IncludeList --> ExcludeClients{Exclude Clients?} AllClients --> ExcludeClients ExcludeClients -->|Yes| RemoveClients[Remove Excluded
MCP Clients] ExcludeClients -->|No| ClientsFiltered[Filtered Clients] RemoveClients --> ToolFilter[Tool-Level Filtering] ClientsFiltered --> ToolFilter ToolFilter --> IncludeTools{Include Tools?} IncludeTools -->|Yes| IncludeSpecific[Include Specified
Tools Only] IncludeTools -->|No| AllTools[All Available Tools] IncludeSpecific --> ExcludeTools{Exclude Tools?} AllTools --> ExcludeTools ExcludeTools -->|Yes| RemoveTools[Remove Excluded
Tools] ExcludeTools -->|No| FinalTools[Final Tool Set] RemoveTools --> FinalTools FinalTools --> AIModel[Available to AI Model] NoMCP --> AIModel ``` ### **Filtering Configuration Levels** **Request-Level Filtering:** ```bash theme={null} # Include only specific MCP clients curl -X POST http://localhost:8080/v1/chat/completions \ -H "x-bf-mcp-include-clients: filesystem,websearch" \ -d '{"model": "gpt-4o-mini", "messages": [...]}' # Include only specific tools curl -X POST http://localhost:8080/v1/chat/completions \ -H "x-bf-mcp-include-tools: filesystem-read_file,websearch-search" \ -d '{"model": "gpt-4o-mini", "messages": [...]}' ``` **Configuration-Level Filtering:** * **Client Selection** - Choose which MCP servers to connect to * **Tool Blacklisting** - Permanently disable dangerous or unwanted tools * **Permission Mapping** - Map user roles to available tool sets * **Environment-Based** - Different tool sets for development vs production **Security Benefits:** * **Principle of Least Privilege** - Only necessary tools are exposed * **Dynamic Access Control** - Per-request tool availability * **Audit Trail** - Track which tools are used by which requests * **Risk Mitigation** - Prevent access to dangerous operations > **📖 Tool Filtering:** [MCP Tool Control →](../../mcp/filtering) *** ## Tool Execution Engine ### **Async Tool Execution Architecture** The MCP execution engine handles tool invocation asynchronously to maintain system responsiveness and enable complex multi-tool workflows: ```mermaid theme={null} sequenceDiagram participant AIModel participant ExecutionEngine participant ToolInvoker participant MCPServer participant ResultProcessor AIModel->>ExecutionEngine: Tool Call Request ExecutionEngine->>ExecutionEngine: Validate Tool Call ExecutionEngine->>ToolInvoker: Queue Tool Execution Note over ToolInvoker: Async Tool Execution ToolInvoker->>MCPServer: Invoke Tool MCPServer->>MCPServer: Execute Tool Logic MCPServer-->>ToolInvoker: Raw Tool Result ToolInvoker->>ResultProcessor: Process Result ResultProcessor->>ResultProcessor: Format & Validate ResultProcessor-->>ExecutionEngine: Processed Result ExecutionEngine-->>AIModel: Tool Execution Complete Note over AIModel: Multi-turn Conversation AIModel->>ExecutionEngine: Continue with Tool Results ExecutionEngine->>ExecutionEngine: Merge Results into Context ExecutionEngine-->>AIModel: Enhanced Response ``` ### **Execution Flow Characteristics** **Validation Phase:** * **Parameter Validation** - Ensure tool arguments match expected schema * **Permission Checking** - Verify tool access permissions for the request * **Rate Limiting** - Apply per-tool and per-user rate limits * **Security Scanning** - Check for potentially dangerous operations **Execution Phase:** * **Timeout Management** - Bounded execution time to prevent hanging * **Error Handling** - Graceful handling of tool failures and timeouts * **Result Streaming** - Support for tools that return streaming responses * **Resource Monitoring** - Track tool resource usage and performance **Response Phase:** * **Result Formatting** - Convert tool outputs to consistent format * **Error Enrichment** - Add context and suggestions for tool failures * **Multi-Result Aggregation** - Combine multiple tool outputs coherently * **Context Integration** - Merge tool results into conversation context ### **Multi-Turn Conversation Support** The MCP system enables sophisticated multi-turn conversations where AI models can: 1. **Initial Tool Discovery** - Request available tools for a given context 2. **Tool Execution** - Execute one or more tools based on user request 3. **Result Analysis** - Analyze tool outputs and determine next steps 4. **Follow-up Actions** - Execute additional tools based on previous results 5. **Response Synthesis** - Combine tool results into coherent user response **Example Multi-Turn Flow:** ``` User: "Find recent news about AI and save interesting articles" AI: → Execute web_search("AI news recent") AI: → Analyze search results AI: → Execute save_article() for each interesting result AI: → Respond with summary of saved articles ``` ### **Complete User-Controlled Tool Execution Flow** The following diagram shows the end-to-end user experience with MCP tool execution, highlighting the critical user control points and decision-making process: ```mermaid theme={null} flowchart TD A["👤 User Message
\"List files in current directory\""] --> B["🤖 Bifrost Core"] B --> C["🔧 MCP Manager
Auto-discovers and adds
available tools to request"] C --> D["🌐 LLM Provider
(OpenAI, Anthropic, etc.)"] D --> E{"🔍 Response contains
tool_calls?"} E -->|No| F["✅ Final Response
Display to user"] E -->|Yes| G["📝 Add assistant message
with tool_calls to history"] G --> H["🛡️ YOUR EXECUTION LOGIC
(Security, Approval, Logging)"] H --> I{"🤔 User Decision Point
Execute this tool?"} I -->|Deny| J["❌ Create denial result
Add to conversation history"] I -->|Approve| K["⚙️ client.ExecuteMCPTool()
Bifrost executes via MCP"] K --> L["📊 Tool Result
Add to conversation history"] J --> M["🔄 Continue conversation loop
Send updated history back to LLM"] L --> M M --> D style A fill:#e1f5fe style F fill:#e8f5e8 style H fill:#fff3e0 style I fill:#fce4ec style K fill:#f3e5f5 ``` **Key Flow Characteristics:** **User Control Points:** * **Security Layer** - Your application controls all tool execution decisions * **Approval Gate** - Users can approve or deny each tool execution * **Transparency** - Full visibility into what tools will be executed and why * **Conversation Continuity** - Tool results seamlessly integrate into conversation flow **Security Benefits:** * **No Automatic Execution** - Tools never execute without explicit approval * **Audit Trail** - Complete logging of all tool execution decisions * **Contextual Security** - Approval decisions can consider full conversation context * **Graceful Denials** - Denied tools result in informative responses, not errors **Implementation Patterns:** ```go theme={null} // Example tool execution control in your application func handleToolExecution(toolCall schemas.ChatToolCall, userContext UserContext) error { // YOUR SECURITY AND APPROVAL LOGIC HERE if !userContext.HasPermission(toolCall.Function.Name) { return createDenialResponse("Tool not permitted for user role") } if requiresApproval(toolCall) { approved := promptUserForApproval(toolCall) if !approved { return createDenialResponse("User denied tool execution") } } // Execute the tool via Bifrost result, err := client.ExecuteMCPTool(ctx, toolCall) if err != nil { return handleToolError(err) } return addToolResultToHistory(result) } ``` This flow ensures that while AI models can discover and request tool usage, all actual execution remains under user control, providing the perfect balance of AI capability and human oversight. *** ## Agent Mode Architecture Agent Mode transforms Bifrost into an autonomous agent runtime by automatically executing pre-approved tools. This section details the internal architecture of the agent execution loop. ### **Agent Execution Loop** The agent mode operates as an iterative loop that continues until one of the termination conditions is met: ```mermaid theme={null} flowchart TD subgraph "Agent Mode Entry" A["📥 Incoming Chat Request"] --> B{"🔍 Check MCP Config
Any tools_to_auto_execute?"} B -->|No| C["📤 Standard Flow
Return tool_calls for manual execution"] B -->|Yes| D["🤖 Enter Agent Loop"] end subgraph "Agent Execution Loop" D --> E["🌐 Send to LLM Provider
With available tools"] E --> F{"🔧 Response has
tool_calls?"} F -->|No| G["✅ Return Final Response
No more tools needed"] F -->|Yes| H["📋 Classify Tool Calls"] H --> I{"🔐 Separate by
auto-execute status"} I --> J["⚡ Auto-Executable Tools"] I --> K["🛡️ Non-Auto-Executable Tools"] J --> L["🔄 Execute in Parallel
Via ToolsManager"] L --> M["📊 Collect Results"] K --> N{"Any non-auto
tools found?"} N -->|Yes| O["🛑 Exit Loop Early
Return mixed response"] N -->|No| P{"⏱️ Max depth
reached?"} M --> P P -->|Yes| Q["⚠️ Return Current State
May have pending tools"] P -->|No| R["📝 Add results to history"] R --> E end subgraph "Response Handling" O --> S["📦 Create Mixed Response
• Content: executed results JSON
• tool_calls: pending tools
• finish_reason: stop"] G --> T["📦 Standard Response
Final answer from LLM"] Q --> U["📦 Depth Limit Response
Current state with any pending"] end style D fill:#e3f2fd style L fill:#e8f5e9 style O fill:#fff3e0 style S fill:#fce4ec ``` ### **Tool Classification System** When the LLM returns tool calls, Bifrost classifies each tool based on the client configuration: ```mermaid theme={null} flowchart LR subgraph "Tool Call Classification" TC["🔧 Tool Call
from LLM Response"] --> CHECK{"Tool in
tools_to_execute?"} CHECK -->|No| SKIP["❌ Skip
Not allowed"] CHECK -->|Yes| AUTO{"Tool in
tools_to_auto_execute?"} AUTO -->|Yes| EXEC["⚡ Auto-Execute
Run immediately"] AUTO -->|No| MANUAL["🛡️ Manual
Return to caller"] end subgraph "Configuration Example" CONFIG["MCPClientConfig"] CONFIG --> TE["tools_to_execute: [*]
All tools available"] CONFIG --> TAE["tools_to_auto_execute:
[read_file, list_dir]"] end style EXEC fill:#c8e6c9 style MANUAL fill:#fff9c4 style SKIP fill:#ffcdd2 ``` ### **Mixed Tool Response Format** When a response contains both auto-executable and non-auto-executable tools, the agent creates a special response format:
Initial Request"] --> D1["Depth 1
First tool execution"] D1 --> D2["Depth 2
Second iteration"] D2 --> D3["Depth 3
..."] D3 --> DN["Depth N
Max reached"] end DN --> EXIT["🛑 Force Exit
Return current state"] subgraph "Configuration" CFG["MCPToolManagerConfig"] CFG --> MAX["max_agent_depth: 10
(default)"] CFG --> TIMEOUT["tool_execution_timeout:
30s per tool"] end ```
Python-like Runtime"] VFS["📁 Virtual File System
Tool Definitions as .pyi"] EXEC["⚙️ Code Executor
Sandboxed Execution"] end subgraph "Meta Tools" LIST["listToolFiles()
Discover available servers"] READ["readToolFile(fileName)
Get tool signatures"] DOCS["getToolDocs(server, tool)
Get detailed docs"] CODE["executeToolCode(code)
Run Python code"] end subgraph "MCP Integration" TOOLS["🔧 Connected MCP Tools"] RESULTS["📊 Tool Results"] end LLM["🤖 LLM"] --> LIST LIST --> VFS VFS --> LLM LLM --> READ READ --> VFS VFS --> LLM LLM --> DOCS DOCS --> VFS VFS --> LLM LLM --> CODE CODE --> VM VM --> EXEC EXEC --> TOOLS TOOLS --> RESULTS RESULTS --> LLM style VM fill:#e8eaf6 style VFS fill:#e3f2fd style CODE fill:#e8f5e9 ``` ### **Virtual File System (VFS)** Code Mode generates Python stub files (`.pyi`) for all connected MCP tools, providing compact function signatures:
Syntax checking before execution"] L2["🛡️ Sandboxed Runtime
No external module access"] L3["⏱️ Execution Timeout
Bounded runtime"] L4["🔐 Tool ACL
Only allowed tools accessible"] end subgraph "Execution Boundaries" B1["No filesystem access
(except via MCP tools)"] B2["No network access
(except via MCP tools)"] B3["No process spawning"] B4["Memory isolation enforced"] end L1 --> L2 --> L3 --> L4 L4 --> B1 L4 --> B2 L4 --> B3 L4 --> B4 ``` ### **Code Mode Configuration**
Authentication & Encryption] L2[Tool Validation
Schema & Permission Checks] L3[Execution Security
Sandboxing & Limits] L4[Result Security
Output Validation & Filtering] end subgraph "Threat Mitigation" T1[Malicious Tools
Code Injection Prevention] T2[Resource Abuse
Rate Limiting & Quotas] T3[Data Exposure
Output Sanitization] T4[System Access
Privilege Isolation] end L1 --> T1 L2 --> T2 L3 --> T4 L4 --> T3 ``` **Security Measures:** **Connection Security:** * **Authentication** - API keys, certificates, or token-based auth for HTTP/SSE * **Encryption** - TLS for HTTP connections, secure pipes for STDIO * **Network Isolation** - Firewall rules and network segmentation **Execution Security:** * **Sandboxing** - Isolated execution environments for tools * **Resource Limits** - CPU, memory, and time constraints * **Permission Model** - Principle of least privilege for tool access **Operational Security:** * **Regular Updates** - Keep MCP servers and tools updated * **Monitoring** - Continuous security monitoring and alerting * **Incident Response** - Procedures for security incidents involving tools *** ## Related Architecture Documentation * **[Request Flow](./request-flow)** - MCP integration in request processing * **[Concurrency Model](./concurrency)** - MCP concurrency and worker integration * **[Plugin System](./plugins)** - Integration between MCP and plugin systems * **[Benchmarks](../../benchmarking/getting-started)** - MCP performance impact and optimization # Plugins Source: https://docs.getbifrost.ai/architecture/core/plugins Deep dive into Bifrost's extensible plugin architecture - how plugins work internally, lifecycle management, execution model, and integration patterns. ## Plugin Architecture Philosophy ### **Core Design Principles** Bifrost's plugin system is built around five key principles that ensure extensibility without compromising performance or reliability: | Principle | Implementation | Benefit | | -------------------------- | ------------------------------------------------ | ------------------------------------------------ | | **Plugin-First Design** | Core logic designed around plugin hook points | Maximum extensibility without core modifications | | **Zero-Copy Integration** | Direct memory access to request/response objects | Minimal performance overhead | | **Lifecycle Management** | Complete plugin lifecycle with automatic cleanup | Resource safety and leak prevention | | **Interface-Based Safety** | Well-defined interfaces for type safety | Compile-time validation and consistency | | **Failure Isolation** | Plugin errors don't crash the core system | Fault tolerance and system stability | ### **Plugin System Overview** ```mermaid theme={null} graph TB subgraph "Plugin Management Layer" PluginMgr[Plugin Manager
Central Controller] Registry[Plugin Registry
Discovery & Loading] Lifecycle[Lifecycle Manager
State Management] end subgraph "Plugin Execution Layer" Pipeline[Plugin Pipeline
Execution Orchestrator] PreHooks[Pre-Processing Hooks
Request Modification] PostHooks[Post-Processing Hooks
Response Enhancement] end subgraph "Plugin Categories" Auth[Authentication
& Authorization] RateLimit[Rate Limiting
& Throttling] Transform[Data Transformation
& Validation] Monitor[Monitoring
& Analytics] Custom[Custom Business
Logic] end PluginMgr --> Registry Registry --> Lifecycle Lifecycle --> Pipeline Pipeline --> PreHooks Pipeline --> PostHooks PreHooks --> Auth PreHooks --> RateLimit PostHooks --> Transform PostHooks --> Monitor PostHooks --> Custom ``` *** ## Plugin Lifecycle Management ### **Complete Lifecycle States** Every plugin goes through a well-defined lifecycle that ensures proper resource management and error handling: ```mermaid theme={null} stateDiagram-v2 [*] --> PluginInit: Plugin Creation PluginInit --> Registered: Add to BifrostConfig Registered --> PreRequestHookCall: Request Received (once per request) PreRequestHookCall --> RouteDecided: Provider/Model resolved PreRequestHookCall --> RouteDecided: Return Error (logged, non-blocking) RouteDecided --> PreHookCall: Per-attempt phase PreHookCall --> ModifyRequest: Normal Flow PreHookCall --> ShortCircuitResponse: Return Response PreHookCall --> ShortCircuitError: Return Error ModifyRequest --> ProviderCall: Send to Provider ProviderCall --> PostHookCall: Receive Response ShortCircuitResponse --> PostHookCall: Skip Provider ShortCircuitError --> PostHookCall: Pipeline Symmetry PostHookCall --> ModifyResponse: Process Result PostHookCall --> RecoverError: Error Recovery PostHookCall --> FallbackCheck: Check AllowFallbacks PostHookCall --> ResponseReady: Pass Through FallbackCheck --> TryFallback: AllowFallbacks=true/nil FallbackCheck --> ResponseReady: AllowFallbacks=false TryFallback --> PreHookCall: Next Provider (PreRequestHook NOT re-run) ModifyResponse --> ResponseReady: Modified RecoverError --> ResponseReady: Recovered ResponseReady --> [*]: Return to Client Registered --> CleanupCall: Bifrost Shutdown CleanupCall --> [*]: Plugin Destroyed ``` ### **Lifecycle Phase Details** **Discovery Phase:** * **Purpose:** Find and catalog available plugins * **Sources:** Command line, environment variables, JSON configuration, directory scanning * **Validation:** Basic existence and format checks * **Output:** Plugin descriptors with metadata **Loading Phase:** * **Purpose:** Load plugin binaries into memory * **Security:** Digital signature verification and checksum validation * **Compatibility:** Interface implementation validation * **Resource:** Memory and capability assessment **Initialization Phase:** * **Purpose:** Configure plugin with runtime settings * **Timeout:** Bounded initialization time to prevent hanging * **Dependencies:** External service connectivity verification * **State:** Internal state setup and resource allocation **Runtime Phase:** * **Purpose:** Active request processing * **Monitoring:** Continuous health checking and performance tracking * **Recovery:** Automatic error recovery and degraded mode handling * **Metrics:** Real-time performance and health metrics collection > **Plugin Lifecycle:** [Plugin Management →](../../plugins/getting-started) *** ## Plugin Execution Pipeline ### **Request Processing Flow** The plugin pipeline ensures consistent, predictable execution while maintaining high performance: #### **Normal Execution Flow (No Short-Circuit)** ```mermaid theme={null} sequenceDiagram participant Client participant Bifrost participant Plugin1 participant Plugin2 participant Provider Client->>Bifrost: Request Note over Bifrost,Plugin2: PreRequestHook phase (once per request, before any fan-out) Bifrost->>Plugin1: PreRequestHook(request) Plugin1-->>Bifrost: routed request Bifrost->>Plugin2: PreRequestHook(request) Plugin2-->>Bifrost: routed request Note over Bifrost,Plugin2: PreLLMHook phase (per provider attempt) Bifrost->>Plugin1: PreLLMHook(request) Plugin1-->>Bifrost: modified request Bifrost->>Plugin2: PreLLMHook(request) Plugin2-->>Bifrost: modified request Bifrost->>Provider: API Call Provider-->>Bifrost: response Bifrost->>Plugin2: PostLLMHook(response) Plugin2-->>Bifrost: modified response Bifrost->>Plugin1: PostLLMHook(response) Plugin1-->>Bifrost: modified response Bifrost-->>Client: Final Response ``` **Execution Order:** 1. **PreRequestHooks** (per-request, registration order 1 → 2 → N): the **routing phase**. Plugins decide which provider/model the request goes to. Mutations to `req.Provider`/`req.Model`/`req.Fallbacks` commit to the shared request and are observed by every subsequent phase and every fallback attempt. There is no short-circuit. Plugin errors are non-blocking — logged as warnings and the pipeline continues to the next plugin. After all PreRequestHooks have run, the core validates `req.Provider`: an unresolved provider returns a 400 to the caller. 2. **PreLLMHooks** (per attempt, registration order 1 → 2 → N): pre-call transforms — caching, validation, content modification. May short-circuit with a synthetic response. 3. **Provider Call:** if no short-circuit occurred. 4. **PostLLMHooks** (per attempt, reverse order N → 2 → 1): response transforms — error recovery, logging, observability. **Per-request vs per-attempt:** `PreRequestHook` runs **exactly once** at the top of `handleRequest` / `handleStreamRequest`, before any provider call. `PreLLMHook` and `PostLLMHook` run **once per provider attempt** — so if the primary call fails and a fallback fires, `PreLLMHook` and `PostLLMHook` run again on the fallback, but `PreRequestHook` does **not**. This is what makes `PreRequestHook` the right place for routing decisions: the decision is committed once and applies uniformly to the primary attempt and every fallback.
AllowFallbacks=false"] D -->|Yes| F["RateLimit Plugin PreLLMHook"] F --> G{"Rate Limited?"} G -->|Yes| H["Return Rate Error
AllowFallbacks=nil"] G -->|No| I["Cache Plugin PreLLMHook"] I --> J{"Cache Hit?"} J -->|Yes| K["Return Cached Response"] J -->|No| L["Provider API Call"] L --> M["Cache Plugin PostLLMHook"] M --> N["Store in Cache"] N --> O["RateLimit Plugin PostLLMHook"] O --> P["Auth Plugin PostLLMHook"] P --> Q["Final Response"] E --> R["Skip Fallbacks"] H --> S["Try Fallback Provider"] K --> T["Skip Provider Call"] ``` ### **Execution Characteristics** **Symmetric Execution Pattern:** * **Pre-processing:** Plugins execute in priority order (high to low) * **Post-processing:** Plugins execute in reverse order (low to high) * **Rationale:** Ensures proper cleanup and state management (last in, first out) **Performance Optimizations:** * **Timeout Boundaries:** Each plugin has configurable execution timeouts * **Panic Recovery:** Plugin panics are caught and logged without crashing the system * **Resource Limits:** Memory and CPU limits prevent runaway plugins * **Circuit Breaking:** Repeated failures trigger plugin isolation **Error Handling Strategies:** * **Continue:** Use original request/response if plugin fails * **Fail Fast:** Return error immediately if critical plugin fails * **Retry:** Attempt plugin execution with exponential backoff * **Fallback:** Use alternative plugin or default behavior > **Plugin Execution:** [Request Flow →](./request-flow#stage-3-plugin-pipeline-processing) *** ## Security & Validation ### **Multi-Layer Security Model** Plugin security operates at multiple layers to ensure system integrity: ```mermaid theme={null} graph TB subgraph "Security Validation Layers" L1[Layer 1: Binary Validation
Signature & Checksum] L2[Layer 2: Interface Validation
Type Safety & Compatibility] L3[Layer 3: Runtime Validation
Resource Limits & Timeouts] L4[Layer 4: Execution Isolation
Panic Recovery & Error Handling] end subgraph "Security Benefits" Integrity[Code Integrity
Verified Authenticity] Safety[Type Safety
Compile-time Checks] Stability[System Stability
Isolated Failures] Performance[Performance Protection
Resource Limits] end L1 --> Integrity L2 --> Safety L3 --> Performance L4 --> Stability ``` ### **Validation Process** **Binary Security:** * **Digital Signatures:** Cryptographic verification of plugin authenticity * **Checksum Validation:** File integrity verification * **Source Verification:** Trusted source requirements **Interface Security:** * **Type Safety:** Interface implementation verification * **Version Compatibility:** Plugin API version checking * **Memory Safety:** Safe memory access patterns **Runtime Security:** * **Resource Quotas:** Memory and CPU usage limits * **Execution Timeouts:** Bounded execution time * **Sandbox Execution:** Isolated execution environment **Operational Security:** * **Health Monitoring:** Continuous plugin health assessment * **Error Tracking:** Plugin error rate monitoring * **Automatic Recovery:** Failed plugin restart and recovery *** ## Plugin Performance & Monitoring ### **Comprehensive Metrics System** Bifrost provides detailed metrics for plugin performance and health monitoring: ```mermaid theme={null} graph TB subgraph "Execution Metrics" ExecTime[Execution Time
Latency per Plugin] ExecCount[Execution Count
Request Volume] SuccessRate[Success Rate
Error Percentage] Throughput[Throughput
Requests/Second] end subgraph "Resource Metrics" MemoryUsage[Memory Usage
Per Plugin Instance] CPUUsage[CPU Utilization
Processing Time] IOMetrics[I/O Operations
Network/Disk Activity] PoolUtilization[Pool Utilization
Resource Efficiency] end subgraph "Health Metrics" ErrorRate[Error Rate
Failed Executions] PanicCount[Panic Recovery
Crash Events] TimeoutCount[Timeout Events
Slow Executions] RecoveryRate[Recovery Success
Failure Handling] end subgraph "Business Metrics" AddedLatency[Added Latency
Plugin Overhead] SystemImpact[System Impact
Overall Performance] FeatureUsage[Feature Usage
Plugin Utilization] CostImpact[Cost Impact
Resource Consumption] end ``` ### **Performance Characteristics** **Plugin Execution Performance:** * **Typical Overhead:** 1-10μs per plugin for simple operations * **Authentication Plugins:** 1-5μs for key validation * **Rate Limiting Plugins:** 500ns for quota checks * **Monitoring Plugins:** 200ns for metric collection * **Transformation Plugins:** 2-10μs depending on complexity **Resource Usage Patterns:** * **Memory Efficiency:** Object pooling reduces allocations * **CPU Optimization:** Minimal processing overhead * **Network Impact:** Configurable external service calls * **Storage Overhead:** Minimal for stateless plugins *** ## Plugin Integration Patterns ### **Common Integration Scenarios** **1. Authentication & Authorization** * **Pre-processing Hook:** Validate API keys or JWT tokens * **Configuration:** External identity provider integration * **Error Handling:** Return 401/403 responses for invalid credentials * **Performance:** Sub-5μs validation with caching **2. Rate Limiting & Quotas** * **Pre-processing Hook:** Check request quotas and limits * **Storage:** Redis or in-memory rate limit tracking * **Algorithms:** Token bucket, sliding window, fixed window * **Responses:** 429 Too Many Requests with retry headers **3. Request/Response Transformation** * **Dual Hooks:** Pre-processing for requests, post-processing for responses * **Use Cases:** Data format conversion, field mapping, content filtering * **Performance:** Streaming transformations for large payloads * **Compatibility:** Provider-specific format adaptations **4. Monitoring & Analytics** * **Post-processing Hook:** Collect metrics and logs after request completion * **Destinations:** Prometheus, DataDog, custom analytics systems * **Data:** Request/response metadata, performance metrics, error tracking * **Privacy:** Configurable data sanitization and filtering ### **Plugin Communication Patterns** **Plugin-to-Plugin Communication:** * **Shared Context:** Plugins can store data in request context for downstream plugins * **Event System:** Plugin can emit events for other plugins to consume * **Data Passing:** Structured data exchange between related plugins **Plugin-to-External Service Communication:** * **HTTP Clients:** Built-in HTTP client pools for external API calls * **Database Connections:** Connection pooling for database access * **Message Queues:** Integration with message queue systems * **Caching Systems:** Redis, Memcached integration for state storage **Plugin-to-Bifrost Communication:** Bifrost exposes internal services to plugins as methods on the request context, so a plugin never has to be handed a service handle at construction time. The context carries a small interface handle; the implementation stays in Bifrost. * **Model catalog:** `ctx.GetModelInfo(provider, model)` returns pricing, context window and capability metadata; `ctx.CalculateCost(resp)` prices a completed response with governance overrides applied * **Tracing:** `ctx.SetTraceAttribute(key, value)` and the stream-control methods delegate to the active tracer * **Logging:** `ctx.Log(level, msg)` writes plugin-scoped entries collected per request Because these handles live on the request context rather than in plugin constructors, they reach plugins compiled as standalone `.so` binaries with no Bifrost framework dependency. Every accessor is inert - returning a zero value rather than panicking - when the backing service is not configured, which keeps a plugin portable between the HTTP gateway and Go SDK embeddings. > **📖 Integration Examples:** [Plugin Development Guide →](../../plugins/writing-go-plugin) *** ## Related Architecture Documentation * **[Request Flow](./request-flow)** - Plugin execution in request processing pipeline * **[Concurrency Model](./concurrency)** - Plugin concurrency and threading considerations * **[Benchmarks](../../benchmarking/getting-started)** - Plugin performance characteristics and optimization * **[MCP System](./mcp)** - Integration between plugins and MCP system # Request Flow Source: https://docs.getbifrost.ai/architecture/core/request-flow Deep dive into Bifrost's request processing pipeline - from transport layer ingestion through provider execution to response delivery. ## Stage 1: Transport Layer Processing ### **HTTP Transport Flow** ```mermaid theme={null} sequenceDiagram participant Client participant HTTPTransport participant Router participant Validation Client->>HTTPTransport: POST /v1/chat/completions HTTPTransport->>HTTPTransport: Parse Headers HTTPTransport->>HTTPTransport: Extract Body HTTPTransport->>Validation: Validate JSON Schema Validation->>Router: BifrostRequest Router-->>HTTPTransport: Processing Started HTTPTransport-->>Client: HTTP 200 (async processing) ``` **Key Processing Steps:** 1. **Request Reception** - FastHTTP server receives request 2. **Header Processing** - Extract authentication, content-type, custom headers 3. **Body Parsing** - JSON unmarshaling with schema validation 4. **Request Transformation** - Convert to internal `BifrostRequest` schema 5. **Context Creation** - Build request context with metadata **Performance Characteristics:** * **Parsing Time:** \~2.1μs for typical requests * **Validation Overhead:** \~400ns for schema checks * **Memory Allocation:** Zero-copy where possible ### **Go SDK Flow** ```mermaid theme={null} sequenceDiagram participant Application participant SDK participant Core participant Validation Application->>SDK: bifrost.ChatCompletion(req) SDK->>SDK: Type Validation SDK->>Core: Direct Function Call Core->>Validation: Schema Validation Validation-->>Core: Validated Request Core-->>SDK: Processing Result SDK-->>Application: Typed Response ``` **Advantages:** * **Zero Serialization** - Direct Go struct passing * **Type Safety** - Compile-time validation * **Lower Latency** - No HTTP/JSON overhead * **Memory Efficiency** - No intermediate allocations *** ## Stage 2: Request Routing & Load Balancing ### **Provider Selection Logic** ```mermaid theme={null} flowchart TD Request[Incoming Request] --> ModelCheck{Model Available?} ModelCheck -->|Yes| ProviderDirect[Use Specified Provider] ModelCheck -->|No| ModelMapping[Model → Provider Mapping] ProviderDirect --> KeyPool[API Key Pool] ModelMapping --> KeyPool KeyPool --> WeightedSelect[Weighted Random Selection] WeightedSelect --> HealthCheck{Provider Healthy?} HealthCheck -->|Yes| AssignWorker[Assign Worker] HealthCheck -->|No| CircuitBreaker[Circuit Breaker] CircuitBreaker --> FallbackCheck{Fallback Available?} FallbackCheck -->|Yes| FallbackProvider[Try Fallback] FallbackCheck -->|No| ErrorResponse[Return Error] FallbackProvider --> KeyPool ``` **Key Selection Algorithm:** ```go theme={null} // Weighted random key selection type KeySelector struct { keys []APIKey weights []float64 total float64 } func (ks *KeySelector) SelectKey() *APIKey { r := rand.Float64() * ks.total cumulative := 0.0 for i, weight := range ks.weights { cumulative += weight if r <= cumulative { return &ks.keys[i] } } return &ks.keys[len(ks.keys)-1] } ``` **Performance Metrics:** * **Key Selection Time:** \~10ns (constant time) * **Health Check Overhead:** \~50ns (cached results) * **Fallback Decision:** \~25ns (configuration lookup) *** ## Stage 3: Plugin Pipeline Processing ### **Pre-Processing Hooks** ```mermaid theme={null} sequenceDiagram participant Request participant AuthPlugin participant RateLimitPlugin participant TransformPlugin participant Core Request->>AuthPlugin: ProcessRequest() AuthPlugin->>AuthPlugin: Validate API Key AuthPlugin->>RateLimitPlugin: Authorized Request RateLimitPlugin->>RateLimitPlugin: Check Rate Limits RateLimitPlugin->>TransformPlugin: Allowed Request TransformPlugin->>TransformPlugin: Modify Request TransformPlugin->>Core: Final Request ``` **Plugin Execution Model:** ```go theme={null} type PluginManager struct { plugins []Plugin } func (pm *PluginManager) ExecutePreHooks( ctx BifrostContext, req *BifrostRequest, ) (*BifrostRequest, *BifrostError) { for _, plugin := range pm.plugins { modifiedReq, err := plugin.ProcessRequest(ctx, req) if err != nil { return nil, err } req = modifiedReq } return req, nil } ``` **Plugin Types & Performance:** | Plugin Type | Processing Time | Memory Impact | Failure Mode | | --------------------- | --------------- | ------------- | ---------------------- | | **Authentication** | \~1-5μs | Minimal | Reject request | | **Rate Limiting** | \~500ns | Cache-based | Throttle/reject | | **Request Transform** | \~2-10μs | Copy-on-write | Continue with original | | **Monitoring** | \~200ns | Append-only | Continue silently | *** ## Stage 4: MCP Tool Discovery & Integration ### **Tool Discovery Process** ```mermaid theme={null} flowchart TD Request[Request with Model] --> MCPCheck{MCP Enabled?} MCPCheck -->|No| SkipMCP[Skip MCP Processing] MCPCheck -->|Yes| ClientLookup[MCP Client Lookup] ClientLookup --> ToolFilter[Tool Filtering] ToolFilter --> ToolInject[Inject Tools into Request] ToolFilter --> IncludeCheck{Include Filter?} ToolFilter --> ExcludeCheck{Exclude Filter?} IncludeCheck -->|Yes| IncludeTools[Include Specified Tools] IncludeCheck -->|No| AllTools[Include All Tools] ExcludeCheck -->|Yes| RemoveTools[Remove Excluded Tools] ExcludeCheck -->|No| KeepFiltered[Keep Filtered Tools] IncludeTools --> ToolInject AllTools --> ToolInject RemoveTools --> ToolInject KeepFiltered --> ToolInject ToolInject --> EnhancedRequest[Request with Tools] SkipMCP --> EnhancedRequest ``` **Tool Integration Algorithm:** ```go theme={null} func (mcpm *MCPManager) EnhanceRequest( ctx BifrostContext, req *BifrostChatRequest, ) (*BifrostRequest, error) { // Extract tool filtering from context includeClients := ctx.GetStringSlice("mcp-include-clients") includeTools := ctx.GetStringSlice("mcp-include-tools") // Get available tools availableTools := mcpm.getAvailableTools(includeClients) // Filter tools filteredTools := mcpm.filterTools(availableTools, includeTools) // Inject into request if req.Params == nil { req.Params = &ChatParameters{} } req.Params.Tools = append(req.Params.Tools, filteredTools...) return req, nil } ``` **MCP Performance Impact:** * **Tool Discovery:** \~100-500μs (cached after first request) * **Tool Filtering:** \~50-200ns per tool * **Request Enhancement:** \~1-5μs depending on tool count *** ## Stage 5: Memory Pool Management ### **Object Pool Lifecycle** ```mermaid theme={null} stateDiagram-v2 [*] --> PoolInit: System Startup PoolInit --> Available: Objects Pre-allocated Available --> Acquired: Request Processing Acquired --> InUse: Object Populated InUse --> Processing: Worker Processing Processing --> Completed: Processing Done Completed --> Reset: Object Cleanup Reset --> Available: Return to Pool Available --> Expansion: Pool Exhaustion Expansion --> Available: New Objects Created Reset --> GC: Pool Full GC --> [*]: Garbage Collection ``` **Memory Pool Implementation:** ```go theme={null} type MemoryPools struct { channelPool sync.Pool messagePool sync.Pool responsePool sync.Pool bufferPool sync.Pool } func (mp *MemoryPools) GetChannel() *ProcessingChannel { if ch := mp.channelPool.Get(); ch != nil { return ch.(*ProcessingChannel) } return NewProcessingChannel() } func (mp *MemoryPools) ReturnChannel(ch *ProcessingChannel) { ch.Reset() // Clear previous data mp.channelPool.Put(ch) } ``` *** ## Stage 6: Worker Pool Processing ### **Worker Assignment & Execution** ```mermaid theme={null} sequenceDiagram participant Queue participant WorkerPool participant Worker participant Provider participant Circuit Queue->>WorkerPool: Enqueue Request WorkerPool->>Worker: Assign Available Worker Worker->>Circuit: Check Circuit Breaker Circuit->>Provider: Forward Request Provider-->>Circuit: Response/Error Circuit->>Circuit: Update Health Metrics Circuit-->>Worker: Provider Response Worker-->>WorkerPool: Release Worker WorkerPool-->>Queue: Request Completed ``` **Worker Pool Architecture:** ```go theme={null} type ProviderWorkerPool struct { workers chan *Worker queue chan *ProcessingJob config WorkerPoolConfig metrics *PoolMetrics } func (pwp *ProviderWorkerPool) ProcessRequest(job *ProcessingJob) { // Get worker from pool worker := <-pwp.workers go func() { defer func() { // Return worker to pool pwp.workers <- worker }() // Process request result := worker.Execute(job) job.ResultChan <- result }() } ``` *** ## Stage 7: Provider API Communication ### **HTTP Request Execution** ```mermaid theme={null} sequenceDiagram participant Worker participant HTTPClient participant Provider participant CircuitBreaker participant Metrics Worker->>HTTPClient: PrepareRequest() HTTPClient->>HTTPClient: Add Headers & Auth HTTPClient->>CircuitBreaker: CheckHealth() CircuitBreaker->>Provider: HTTP Request Provider-->>CircuitBreaker: HTTP Response CircuitBreaker->>Metrics: Record Metrics CircuitBreaker-->>HTTPClient: Response/Error HTTPClient-->>Worker: Parsed Response ``` **Request Preparation Pipeline:** ```go theme={null} func (w *ProviderWorker) ExecuteRequest(job *ProcessingJob) *ProviderResponse { // Prepare HTTP request httpReq := w.prepareHTTPRequest(job.Request) // Add authentication w.addAuthentication(httpReq, job.APIKey) // Execute with timeout ctx, cancel := context.WithTimeout(context.Background(), job.Timeout) defer cancel() httpResp, err := w.httpClient.Do(httpReq.WithContext(ctx)) if err != nil { return w.handleError(err, job) } // Parse response return w.parseResponse(httpResp, job) } ``` *** ## Stage 8: Tool Execution & Response Processing ### **MCP Tool Execution Flow** ```mermaid theme={null} sequenceDiagram participant Provider participant MCPProcessor participant MCPServer participant ToolExecutor participant ResponseBuilder Provider->>MCPProcessor: Response with Tool Calls MCPProcessor->>MCPProcessor: Extract Tool Calls loop For each tool call MCPProcessor->>MCPServer: Execute Tool MCPServer->>ToolExecutor: Tool Invocation ToolExecutor-->>MCPServer: Tool Result MCPServer-->>MCPProcessor: Tool Response end MCPProcessor->>ResponseBuilder: Combine Results ResponseBuilder-->>Provider: Enhanced Response ``` **Tool Execution Pipeline:** ```go theme={null} func (mcp *MCPProcessor) ProcessToolCalls( response *ProviderResponse, ) (*ProviderResponse, error) { toolCalls := mcp.extractToolCalls(response) if len(toolCalls) == 0 { return response, nil } // Execute tools concurrently results := make(chan ToolResult, len(toolCalls)) for _, toolCall := range toolCalls { go func(tc ToolCall) { result := mcp.executeTool(tc) results <- result }(toolCall) } // Collect results toolResults := make([]ToolResult, 0, len(toolCalls)) for i := 0; i < len(toolCalls); i++ { toolResults = append(toolResults, <-results) } // Enhance response return mcp.enhanceResponse(response, toolResults), nil } ``` *** ## Stage 9: Post-Processing & Response Formation ### **Plugin Post-Processing** ```mermaid theme={null} sequenceDiagram participant CoreResponse participant LoggingPlugin participant CachePlugin participant MetricsPlugin participant Transport CoreResponse->>LoggingPlugin: ProcessResponse() LoggingPlugin->>LoggingPlugin: Log Request/Response LoggingPlugin->>CachePlugin: Response + Logs CachePlugin->>CachePlugin: Cache Response CachePlugin->>MetricsPlugin: Cached Response MetricsPlugin->>MetricsPlugin: Record Metrics MetricsPlugin->>Transport: Final Response ``` **Response Enhancement Pipeline:** ```go theme={null} func (pm *PluginManager) ExecutePostHooks( ctx BifrostContext, req *BifrostRequest, resp *BifrostResponse, ) (*BifrostResponse, error) { for _, plugin := range pm.plugins { enhancedResp, err := plugin.ProcessResponse(ctx, req, resp) if err != nil { // Log error but continue processing pm.logger.Warn("Plugin post-processing error", "plugin", plugin.Name(), "error", err) continue } resp = enhancedResp } return resp, nil } ``` ### **Response Serialization** ```mermaid theme={null} flowchart TD Response[BifrostResponse] --> Format{Response Format} Format -->|HTTP| JSONSerialize[JSON Serialization] Format -->|SDK| DirectReturn[Direct Go Struct] JSONSerialize --> Compress[Compression] DirectReturn --> TypeCheck[Type Validation] Compress --> Headers[Set Headers] TypeCheck --> Return[Return Response] Headers --> HTTPResponse[HTTP Response] HTTPResponse --> Client[Client Response] Return --> Client ``` *** ## Related Architecture Documentation * **[Concurrency Model](./concurrency)** - Worker pools and threading details * **[Plugin System](./plugins)** - Plugin execution and lifecycle * **[MCP System](./mcp)** - Tool discovery and execution internals * **[Benchmarks](../../benchmarking/getting-started)** - Detailed performance analysis # Config Store Source: https://docs.getbifrost.ai/architecture/framework/config-store A persistent and flexible configuration management system for Bifrost, supporting multiple database backends. The ConfigStore is a critical component of the Bifrost framework, providing a centralized and persistent storage solution for all gateway configurations. It abstracts the underlying database, offering a unified API for managing everything from provider settings and virtual keys to governance policies and plugin configurations. ## Core Features * **Unified Configuration API**: A single interface (`ConfigStore`) for all configuration CRUD (Create, Read, Update, Delete) operations. * **Multiple Backend Support**: Out-of-the-box support for SQLite and PostgreSQL, with an extensible architecture for adding new database backends. * **Comprehensive Data Management**: Manages a wide range of configuration data, including: * Provider and key settings * Virtual keys and governance rules (budgets, rate limits) * Customer and team information for multi-tenancy * Plugin configurations * Vector store and log store settings * Model pricing information * **Transactional Operations**: Ensures data consistency by supporting atomic transactions for complex configuration changes. * **Database Migrations**: Integrated migration system to manage schema evolution across different versions of Bifrost. * **Environment Variable Handling**: Securely manages sensitive data like API keys by storing references to environment variables instead of raw values. ## Architecture The ConfigStore is designed around the `ConfigStore` interface, which defines all the methods for interacting with the configuration data. The primary implementation is `RDBConfigStore`, which uses [GORM](https://gorm.io/) as an ORM to communicate with relational databases. ### Supported Backends * **SQLite**: The default, file-based database, perfect for local development, testing, and single-node deployments. It requires no external services. * **PostgreSQL**: A robust, production-grade database suitable for large-scale, high-availability deployments. The backend is selected and configured in Bifrost's main configuration file. ### Initialization The ConfigStore is initialized at startup based on the provided configuration. ```go theme={null} import ( "github.com/maximhq/bifrost/framework/configstore" "github.com/maximhq/bifrost/core/schemas" ) // Example: Initialize a SQLite-based ConfigStore config := &configstore.Config{ Enabled: true, Type: configstore.ConfigStoreTypeSQLite, Config: &configstore.SQLiteConfig{ File: "/path/to/config.db", }, } var logger schemas.Logger // Assume logger is initialized store, err := configstore.NewConfigStore(context.Background(), config, logger) if err != nil { // Handle error } ``` Here is an example for initializing a PostgreSQL-based `ConfigStore`: ```go theme={null} // Example: Initialize a PostgreSQL-based ConfigStore pgConfig := &configstore.Config{ Enabled: true, Type: configstore.ConfigStoreTypePostgres, Config: &configstore.PostgresConfig{ Host: "localhost", Port: "5432", User: "postgres", Password: "secret", DBName: "bifrost", SSLMode: "disable", MaxIdleConns: 5, // Optional: Maximum idle connections (default: 5) MaxOpenConns: 50, // Optional: Maximum open connections (default: 50) }, } store, err = configstore.NewConfigStore(context.Background(), pgConfig, logger) if err != nil { // Handle error } ```
/virtua-key/quota was not sending back budgets. This is fixed in v1.4.8.
{/* implementation */}
);
};
```
## Testing Conventions
### Go Tests
1. **Test file naming**: `*_test.go`
2. **Test function naming**: `TestBifrost Images] end KSA -->|Annotated with| IAMRole Pod -->|Assumes| IAMRole IAMRole -->|Pull Permission| ECR ECR -->|Image| Pod ``` ## Prerequisites * EKS cluster (v1.23+) or ECS cluster * AWS CLI configured with appropriate permissions * `kubectl` configured for your EKS cluster * Your AWS Account ID allowlisted by Bifrost team
Federation Pool] GSA[GCP Service Account] AR[Artifact Registry
Bifrost Images] end KSA -->|Federated| MI MI -->|OIDC Token| WIF WIF -->|Exchange| GSA GSA -->|Pull Permission| AR AR -->|Image| Pod ``` ## How It Works Azure Workload Identity Federation allows Azure Managed Identities to authenticate to GCP without exchanging credentials: 1. **AKS Pod** requests a token using its Kubernetes ServiceAccount 2. **Azure AD** issues an OIDC token for the Managed Identity 3. **GCP Workload Identity Federation** validates the Azure token 4. **GCP STS** exchanges it for a GCP access token 5. **Pod** uses the GCP token to pull images from Artifact Registry ## Prerequisites * AKS cluster (v1.24+) with Workload Identity enabled * Azure CLI configured with appropriate permissions * `kubectl` configured for your AKS cluster * Your Azure Tenant ID and Managed Identity Client ID provided to Bifrost team
Bifrost Images] end KSA -->|Workload Identity| GSA Pod -->|Impersonates| GSA GSA -->|Pull Permission| AR AR -->|Image| Pod ``` ## Prerequisites * GKE cluster (v1.24+) with Workload Identity enabled * `gcloud` CLI configured with appropriate permissions * `kubectl` configured for your GKE cluster * Your GCP project allowlisted by Bifrost team
Bifrost Images] end Secret -->|Credentials| Pod Pod -->|Pull| AR Docker -->|Pull| AR AR -->|Image| Pod AR -->|Image| Docker ``` ## Prerequisites * Kubernetes cluster (v1.23+) or Docker runtime * Network access to `us-central1-docker.pkg.dev` (or your designated region) * Docker credentials provided by Bifrost team
Claude Desktop and Claude Code
ChatGPT app and ChatGPT web
Codex, Cursor, Gemini
MCP servers in each app"] Edge["Bifrost Edge"] Apps --> Edge end subgraph Company["Your Bifrost"] GW["Gateway and governance
virtual keys and budgets
audit logs and guardrails"] end Edge -- "routed AI traffic" --> GW GW -- "policies, app and MCP allow / deny, config sync" --> Edge ``` ## Why Edge
PII, secrets, content safety, policy] C -- allowed --> D[Model provider] C -- blocked or redacted --> A ``` ## Guardrails that apply out of the box Your existing guardrail profiles cover Edge traffic with no additional configuration. Set them up once in [Guardrails](/enterprise/guardrails) and they take effect across the fleet.
template] -->|copy to user| B[User Access Profile
per-user copy] B -->|materialize| C[Virtual Key
auto-issued] D[Role attached
as default] -->|on user role change| A E[Propagate] -->|sync selected fields| B B -->|regenerate| C ``` 1. **Template** - The Access Profile is the policy you author. Each policy lives once in the workspace. 2. **How it is granted** - A user gets a profile in one of three ways: you attach it to them by hand, their role grants it, or their identity provider attributes match a mapping rule. Bifrost remembers which, so removing one grant never disturbs the others. 3. **User copy** - Bifrost copies the template to the user, with its own budget and rate-limit counters. Each user's usage is tracked separately, and a user can hold several profiles at once. 4. **Virtual key** - Bifrost issues a virtual key to the user. The key belongs to the user rather than to one profile, so it keeps working as their profiles change, and what it may do comes from whichever profiles they currently hold. It is locked against direct edits so the policy cannot be worked around. 5. **Per-model limits** - Per-model budgets set on the profile appear for each assigned user as their own read-only limit. See [Model Limits](/features/governance/model-limits#per-user-model-limits-enterprise). ### Multiple profiles per user A user can hold more than one access profile at a time - one from their role, say, and another attached by hand. The profiles do not merge into a single blended policy. Instead: **Access adds up.** The user can use any provider and any model that *any* of their profiles allows. Giving someone an extra profile can only widen what they can reach; to narrow it, remove a profile or tighten the template. **One profile pays for each request.** Bifrost picks one of the profiles that allows the request and charges it - its budget, its rate limit, and any per-model limit it sets. The user's other profiles are not charged, and their usage is unaffected. **If one profile is out of budget, another covers the request.** A user whose Engineering budget is spent keeps working if another of their profiles can still fund the request. Only when every profile that allows the request has run out is the request blocked, and the message names the profile that was tried. **Usage is tracked per profile.** Each profile has its own counters, so spending against one never eats into another's remaining budget.
model: gpt-4"] subgraph DirectionSelection["Direction Selection"] DS["Provider Selector
Score-based selection"] DP1["OpenAI
score: 0.92"] DP2["Azure
score: 0.85"] DP3["Anthropic
score: 0.78"] end subgraph RouteSelection["Route Selection"] RS["Key Selector
Weighted random"] K1["Key 1
weight: 850"] K2["Key 2
weight: 620"] K3["Key 3
weight: 45"] end subgraph Tracker["Metrics Tracker"] T["Real-time Metrics
5-second recomputation"] M1["Error Rate"] M2["Latency Score"] M3["Utilization"] end Request --> DS DS --> DP1 & DP2 & DP3 DP1 --> RS RS --> K1 & K2 & K3 K1 --> Response["API Response"] Response --> T T --> M1 & M2 & M3 M1 & M2 & M3 -.->|"Update Weights"| DS & RS ``` *** ## How Weight Calculation Works Every 5 seconds, the system recalculates a weight for each route from its recent performance. Three signals drive the score, in priority order: | Factor | Role | Purpose | | ----------------- | --------- | -------------------------------------------------------------------------------- | | **Error Penalty** | Primary | Penalizes routes with high error rates | | **Latency Score** | Secondary | Penalizes routes that are slow relative to their peers and to their own baseline | | **Utilization** | Tuning | Discourages overloading any single high-performing route | Which signals apply depends on the route's health: healthy routes are scored mainly on errors and latency, while routes that are actively recovering are scored on latency and recovery progress so they aren't held back by stale error history. The combined score maps to a weight on a fixed scale - lower penalties mean higher weight, which means more traffic - with a floor so no route is ever fully starved while it has a chance to recover. ```mermaid theme={null} flowchart LR subgraph Inputs["Raw Metrics"] E["Error Rate"] L["Latency"] U["Utilization"] end subgraph Scoring["Health-aware Scoring"] EP["Error Penalty
primary"] LP["Latency Score
peer + baseline"] US["Utilization
balancing"] end subgraph Output["Final Weight"] NS["Combined Score"] FW["Route Weight"] end E --> EP L --> LP U --> US EP & LP & US --> NS NS --> FW ``` *** ## Key Capabilities 1. **Automatic Route Health Management**: Routes automatically transition between 4 states (Healthy, Degraded, Failed, Recovering) based on error rates and latency. No manual intervention required when a route fails or recovers. 2. **Fair Traffic Distribution**: The system prevents any single route from being overloaded while still favoring better performers. Low-weight routes always get minimum traffic to prove recovery. 3. **Real-time Dashboard**: Provides visibility into weight distribution, performance metrics (error rates, latency), state transitions, and actual vs expected traffic per route.
(Cloud Run)"] -->|outbound stream| B["Broker
(relay)"] C["Node B
(Cloud Run)"] -->|outbound stream| B D["Node C
(Cloud Run)"] -->|outbound stream| B B -.->|fan-out| A B -.->|fan-out| C B -.->|fan-out| D ``` A message from Node A travels to the broker, which forwards it to Node B and Node C (never back to A). Because nodes only need **outbound** connectivity, broker mode runs on any platform that can make an outbound gRPC connection. ### How it differs from mesh mode | Aspect | Mesh mode | Broker mode | | ------------------ | ------------------------------------ | ----------------------------------------------------- | | Connectivity | Every node connects to every node | Each node makes one outbound connection to the broker | | Membership | memberlist gossip | Roster pushed by the broker | | Discovery | 6 discovery methods | Not used - the broker is the rendezvous point | | Ports on a node | `10101/TCP+UDP`, `10102/TCP` inbound | None - outbound only | | Leader election | Deterministic over gossip members | Deterministic over the broker roster (same algorithm) | | Entity replication | Over the gRPC mesh | Over the broker relay - identical entity types | ### Broker-mode synchronization performance Broker mode separates cluster traffic into dedicated lanes for general messages, heartbeats, governance usage, KV-store updates, circuit-breaker events, load-balancer logs, and diagnostics. This prevents heavy traffic in one lane from blocking a different kind of cluster traffic. However, all messages of the same type addressed to a node share that node's corresponding broker stream: ```text theme={null} Node B ─┐ Node C ─┼──> Node A's governance stream Node D ─┘ ``` Mesh mode instead has a separate stream between each pair of nodes. As a result, slow processing or heavy traffic from one peer can delay same-lane messages from other peers in broker mode. Broker mode may therefore synchronize more slowly than mesh mode under contention. Use mesh mode when the environment permits direct node-to-node connectivity and maximum synchronization performance is required. Leadership in broker mode uses the **same deterministic rule** as mesh mode: the lexicographically-smallest node ID in the roster is the leader. Every node computes this independently from the roster the broker pushes, so there is nothing to configure and no broker-side election. ### Configuration Nodes run in broker mode by setting `cluster_config.type` to `broker` and pointing at the broker address: ```json theme={null} { "cluster_config": { "enabled": true, "type": "broker", "region": "us-east-1", "broker": { "address": "broker.example.run.app:443", "tls": true, "auth_token": "your-shared-secret" } } } ``` See the [config.json cluster reference](/deployment-guides/config-json/cluster#broker-mode) for the full field list. ### Running the broker The broker is **not** a separate binary - the same Bifrost Enterprise image runs as the broker when started with the `-mode=broker` flag (or the `BIFROST_MODE=broker` environment variable): ```bash theme={null} bifrost-enterprise -mode=broker -app-dir /app/data ``` In broker mode the process branches before the normal server bootstrap: it starts **only** the relay gRPC server and runs no database, providers, plugins, or HTTP gateway. It reads `cluster_config.broker` from the same `config.json` and serves on `broker.listen_port` (default `50051`). A standard gRPC health service is registered for readiness probes. ### Deploying on Cloud Run
Google Cloud Platform
Amazon Web Services
Microsoft Azure
Cloudflare
Vercel
(etcd / Consul / global DNS)"}} subgraph RegionA["Region A"] BA["3 x Bifrost pods"] PGA[("PG primary")] BA -->|"boot read"| PGA end subgraph RegionB["Region B"] BB["3 x Bifrost pods"] PGB[("PG replica")] BB -->|"boot read"| PGB end BA <-.->|"register / discover"| DISC BB <-.->|"register / discover"| DISC PGA -.->|"streaming replication"| PGB BB -.->|"async writes to primary"| PGA ``` Each region serves its own client traffic. Region B pods boot from the local PG replica (read-only is fine for boot), and async writes route to the primary in Region A. RPO is bounded by your replication lag; cluster sync keeps governance counters convergent across regions over the gossip + gRPC transports. ### Geographically split clients, single PG ```mermaid theme={null} flowchart TB PG[("PostgreSQL
(any region)")] EU["Pods - EU"] US["Pods - US"] APAC["Pods - APAC"] EU -->|"boot read"| PG US -->|"boot read"| PG APAC -->|"boot read"| PG EU -.->|"async writes"| PG US -.->|"async writes"| PG APAC -.->|"async writes"| PG ``` A single PostgreSQL primary serves all regions. Pod boot pulls config once across the WAN (acceptable because it is infrequent); steady-state inference traffic never crosses the WAN to hit the DB. Async writes do cross the WAN, which is fine as long as the write queue can absorb it. ## Service discovery is the critical piece Cross-region clustering requires a discovery layer that pods can reach **from every region** to find their peers and join the gossip mesh. Single-region defaults like Kubernetes-scoped discovery, UDP broadcast, or mDNS cannot cross region boundaries - they're physically scoped to one cluster or one broadcast domain. Bifrost ships [six discovery methods](/enterprise/clustering); for cross-region deployments only three of them work: | Method | Why it fits cross-region | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **etcd** | Strong consistency, globally reachable from every region, leases automatically reap dead nodes. Run a multi-region etcd cluster (or a single highly-available etcd) as the registry. See [etcd discovery](/enterprise/clustering#etcd-discovery). | | **Consul** | HashiCorp Consul's multi-datacenter federation is a natural fit: each region registers against the local Consul, federation propagates membership across DCs. See [Consul discovery](/enterprise/clustering#consul-discovery). | | **DNS** | Works if you publish a globally-resolvable SRV or A record covering all regions (e.g., Route 53 cross-region). Simpler than etcd/Consul but offers no health-check loop of its own; dead pods linger in DNS until TTL expires. |
4 vCPU / 16 GB each"] OS["Object Storage
(S3 / GCS / ABS)"] PG[("PostgreSQL
8 vCPU / 16 GB
+ hot standby")] B -->|"large log payloads"| OS B -->|"metadata + small rows"| PG ``` If you start without object storage, bump PostgreSQL to **8 vCPU / 24 GB**; everything else stays the same. # Versioning Source: https://docs.getbifrost.ai/enterprise/moving-from-oss/versioning Why Enterprise and OSS use independent version numbers, and how to find the OSS base each Enterprise release is built on. Bifrost ships two release trains that share code but not version numbers. Read this page once when you start working with Enterprise - it prevents the single most common version-pinning mistake. ## Two independent tracks | Track | Version format | Latest example | Cadence | Source of truth | | -------------- | -------------------- | -------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | **OSS** | `vMAJOR.MINOR.PATCH` | `v1.5.4` | Patches every 2 - 3 days, minor every 3 - 4 weeks | [OSS release cadence](/release-cadence), [OSS changelogs](/changelogs/v1.5.4) | | **Enterprise** | `vMAJOR.MINOR.PATCH` | `v1.4.3` | Same cadence, independent numbering | [Enterprise release cadence](/enterprise/release-cadence), [Enterprise changelogs](/changelogs/ent-v1.4.3) | Both tracks use the **same `vX.Y.Z` format**, but the numbers are on independent counters. There is no digit-for-digit correspondence between Enterprise `v1.4.3` and OSS `v1.4.3` (or any other version pair). Always qualify the version with "Enterprise" or "OSS" in writing so readers know which track you mean.
x-bf-project-id or -name] --> B{Project exists?} B -- no --> R["403 access_blocked
project not found"] B -- yes --> C{Caller admitted?
member, or open membership} C -- no --> R C -- yes --> D{Enabled and not expired?} D -- no --> E["403 access_blocked
project has expired"] D -- yes --> F[Combine access
per the access rule] F --> G[Select which caps apply
per the accounting mode] G --> H[Check caps, serve, then charge] ``` ### Activating a project on a request A request names a project with one of two headers: | Header | Value | | ------------------- | -------------------------------------------- | | `x-bf-project-id` | The project's id | | `x-bf-project-name` | The project's name, which is globally unique | The **Use project** button on a project's page shows both, with a copy-ready request that sends one of them.
headers present?} B -->|Yes| C[Continue to
governance evaluation] B -->|No| D[400 Bad Request
missing_required_headers] ``` When a request arrives: 1. The HTTP transport middleware stores all request headers in the Bifrost context (lowercased keys) 2. The governance plugin's `PreLLMHook` / `PreMCPHook` checks for each required header 3. If any are missing, the request is rejected immediately with a `400` status and a JSON error listing the missing headers **Example error response:** ```json theme={null} { "error": { "message": "missing required headers: x-tenant-id, x-correlation-id", "type": "missing_required_headers" } } ``` *** ## Configuration
parse] B --> C[Middleware
auth] C --> D[Processing
setup + pre-hooks] D --> E[Plugins] E --> F[Key selection] F --> G[Conversion +
Serialization
encode] G --> H[Networking
sign + call] H --> I[[Upstream
provider]] I --> J[Serialization
parse + Conversion] J --> K[Processing
post-hooks] K --> L[Serialization
encode + Client delivery] L --> M[Client response] ``` Two categories are **residuals**: they account for overhead that is not attributed to any single phase (see [The two residuals](#the-two-residuals)). *** ## The categories The breakdown groups its rows into nine categories. Each table below lists every row in a category by the name shown in the drill-down and what it measures. ### Serialization JSON parsing and encoding at the edges of the request. | Name | What it measures | | --------------- | -------------------------------------------------------------------------- | | Request parse | Decoding the incoming client request body into Bifrost's request struct | | Request encode | Encoding the provider-shaped request into JSON bytes for the upstream call | | Response parse | Decoding the provider's raw JSON response into a provider response struct | | Response encode | Encoding the final Bifrost response back to JSON for the client | ### Conversion Translating between Bifrost's unified schema and a provider's native shape. | Name | What it measures | | ------------------------- | ----------------------------------------------------------------------------- | | Schema conversion | Mapping the unified request/response to and from the provider's native format | | Stream convert (inbound) | Per-chunk mapping of provider chunks into the unified shape (streaming only) | | Stream convert (outbound) | Per-chunk mapping of unified chunks into the client's shape (streaming only) | ### Plugins Time spent inside each configured plugin's hooks. One row per plugin, shown by the plugin's name (for example, **Enterprise Governance**, **Semantic Cache**, **OpenTelemetry**), collapsing that plugin's individual hook phases (pre-hook, post-hook) into a single row. Any plugin you configure appears here automatically. ### Middleware HTTP transport authentication and access control, run before the request enters the core pipeline. | Name | What it measures | | ---- | ------------------------------- | | API | API-key validation | | SCIM | SCIM identity resolution | | Auth | Session / access-control checks | ### Key selection Choosing which provider API key to use for the request. | Name | What it measures | | ------------- | ------------------------------------------------- | | Key selection | The weighted pick of a specific key from the pool | | Key pool | Locating the key pool for the resolved provider |
cache markers?} B -- Yes --> F[Forward unchanged] B -- No --> C{prompt_cache
enabled on provider?} C -- No --> F C -- Yes --> D{Model supports
explicit caching?} D -- No --> F D -- Yes --> E[Mark first cacheable block
or configured injection points] E --> G[Provider translates marker
cache_control / cachePoint / prompt_cache_breakpoint] ``` Injection runs on Chat Completions and Responses requests, including their streaming variants, and on every SDK integration route that Bifrost converts into one of those two shapes. A few rules govern what gets marked: 1. **First cacheable block.** With `auto_inject` alone, Bifrost walks the messages in order and marks the first text, image, or file block it finds. A message whose content is a plain string is promoted to a single text block so the marker has somewhere to sit. The promotion is deterministic, so the cached prefix stays byte-identical across turns. 2. **Caller markers win.** If any message already carries a marker, the request is left completely alone. Injection is a default for clients that say nothing, never an override of a client that spoke. 3. **At most four markers.** Anthropic rejects a request carrying more than four blocks with `cache_control`, and every other dialect derives from that ceiling. Injection stops at four rather than relying on a downstream clamp that would silently discard the earliest marker. 4. **Copy on write.** The request Bifrost holds is never mutated. The marker is added to a copy handed to the provider, so plugins, retries, and fallbacks never see a marker the caller did not send. 5. **Per attempt.** A fallback to a different provider re-evaluates injection against that provider's own `prompt_cache` config and model capabilities. It does not inherit the previous provider's decision. *** ## Provider support Bifrost injects one internal marker shape and each provider translates it to its own wire format. The capability gate is evaluated per model, not per provider, so a provider that serves both explicit-caching and implicit-caching models only injects on the former. | Provider | Models that take a marker | Wire format | TTL | | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | Anthropic | All Claude models | `cache_control: {"type": "ephemeral"}` on the content block | `5m` default, `1h` with `ttl` | | Vertex AI | Claude models only. Gemini caches through a server-side `cachedContent` resource, so injection is a no-op there | `cache_control` | `5m` default, `1h` with `ttl` | | Bedrock | Claude and Amazon Nova on the Converse API. gpt-5.6 ids that resolve to the Mantle surface follow the OpenAI row | `cachePoint` block | Claude only. Nova accepts only the default and returns 400 for `1h` | | Bedrock Mantle | Claude, Amazon Nova, and the gpt-5.6 family | `cachePoint` for Claude and Nova, `prompt_cache_breakpoint` for gpt-5.6 | Same as Bedrock | | OpenAI | gpt-5.6 family on the Responses API. Earlier models cache implicitly, and Chat Completions strips the marker for every model | `prompt_cache_breakpoint` on the block plus `prompt_cache_options.mode: "explicit"` on the request | Ignored | | Azure OpenAI | gpt-5.6 family on the Responses API | Same as OpenAI | Ignored | | OpenRouter | Claude models | `cache_control` (Chat) or `prompt_cache_breakpoint` (Responses), converted upstream | `5m` default, `1h` with `ttl` | | Custom providers | Follow the base provider they wrap | Same as the base provider | Same as the base provider | | Gemini, DeepSeek, Groq, xAI, Mistral, and other implicit-caching providers | None | Not applicable, injection is a no-op | Not applicable |
copy x-bf-prompt-id / x-bf-prompt-version to context"] PreHook --> PreLLM["PreLLM hook:
resolve version, merge params,
prepend template messages"] PreLLM --> Provider[Provider] ``` 1. **Transport (HTTP):** Incoming headers `x-bf-prompt-id` and `x-bf-prompt-version` are copied onto the Bifrost context (header name matching is case-insensitive). 2. **Resolve:** The plugin looks up the prompt and the requested version. If **`x-bf-prompt-version` is omitted**, the prompt’s **latest committed version** is used. 3. **Parameters:** Version `model` parameters are merged into the request; any field already set on the request wins. 4. **Messages:** Messages from the committed version are **prepended** to `messages` (chat) or `input` (responses). Your request body adds the user turn(s) after the template. If the prompt ID is missing, the plugin does nothing and the request passes through unchanged. *** ## HTTP headers (gateway) | Header | Required | Description | | --------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------- | | `x-bf-prompt-id` | Yes, to enable injection | UUID of the prompt in the repository. | | `x-bf-prompt-version` | No | **Integer version number** (e.g. `3` for v3). If omitted, the **latest** committed version for that prompt is used. | Invalid or unknown IDs / versions are logged as warnings; the request is **not** failed by the plugin (it proceeds without template injection). *** ## Example: Chat Completions Use the same JSON body as a normal chat request. Only the headers select the template. ```bash theme={null} curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "x-bf-prompt-id: YOUR-PROMPT-UUID" \ -H "x-bf-vk: sk-bf-your-virtual-key" \ -d '{ "model": "openai/gpt-5.4", "messages": [ { "role": "user", "content": "Tell me about Bifrost Gateway?" } ] }' ```
/ / / / / /); evaluating N configured fallback(s)` |
| Each fallback iteration | Info | `Trying fallback i/N: )` |
| Fallback skipped (no provider config) | Warn | `Fallback ); halting further fallbacks` |
| All fallbacks exhausted | Error | `All N fallback(s) exhausted; returning primary error ()` |
| Retry transition (rotated key) | Info | `Retry n/N for ; rotated key=