Clustering - Bifrost

Overview

Bifrost Clustering delivers production-ready high availability through a peer-to-peer network architecture with automatic service discovery. The clustering system uses gossip protocols to maintain consistent state across nodes while providing seamless scaling, automatic failover, and zero-downtime deployments.

Why Clustering Matters

Modern AI gateway deployments require robust infrastructure to handle production workloads:

Challenge	Impact	Clustering Solution
Single Point of Failure	Complete service outage if gateway fails	Distributed architecture with automatic failover
Traffic Spikes	Performance degradation under high load	Dynamic load distribution across multiple nodes
Provider Rate Limits	Request throttling and service interruption	Distributed rate limit tracking across cluster
Regional Latency	Poor user experience in distant regions	Geographic distribution with local processing
Maintenance Windows	Service downtime during updates	Rolling updates with zero-downtime deployment
Capacity Planning	Over/under-provisioning resources	Elastic scaling based on real-time demand

Core Features

Feature	Description
Automatic Service Discovery	6 discovery methods for any infrastructure (K8s, Consul, etcd, DNS, UDP, mDNS)
Peer-to-Peer Architecture	No single point of failure with equal node participation
Gossip-Based State Sync	Real-time synchronization of traffic patterns and limits
Automatic Failover	Seamless traffic redistribution when nodes fail
Zero-Downtime Updates	Rolling deployments without service interruption

Architecture

Peer-to-Peer Network Design

Bifrost clustering uses a peer-to-peer (P2P) network where all nodes are equal participants. Each node:

Discovers peers automatically using the configured discovery method
Receives application state and counter updates over gRPC
Tracks cluster membership and node liveness over a memberlist gossip layer
Handles failover automatically

Cluster Communication

Bifrost uses two transports for different responsibilities. Membership and node-liveness signals run over a memberlist gossip layer; everything else (configuration changes, governance counters, routing rules, all replicated entity types) travels over a dedicated gRPC channel.

Transport	Default port	Carries
Memberlist gossip	`10101` (TCP + UDP)	Cluster membership, node join/leave, liveness probes, region metadata
gRPC counter sync	`10102` (TCP)	Application messages: governance usage counters, config sync, routing rules, virtual keys, providers, RBAC, MCP tools, pricing, auth config, and 25+ other replicated entity types

This split lets membership churn (joins, leaves, failure detection) stay isolated from the higher-volume application message stream, and lets each transport be tuned, scaled, and observed independently. The gRPC layer was introduced in v1.4.0; before then, all traffic ran over gossip.

Application messages and entity types

Each replicated message carries an EntityType identifying the kind of state being broadcast. Bifrost replicates 30+ entity types across the cluster, including: model catalog, virtual keys, providers, governance counters, routing rules, RBAC, MCP tools and tool groups, pricing and pricing overrides, access profiles, prompt deployments, auth configuration, and cluster diagnostics. See the Replicated Entity Types reference for the complete list.

Message dedup and invalidation

Each broadcast carries a unique message ID and a SentAt timestamp. Receivers run a deduper (default 5-minute TTL) keyed by message ID, so a node that has already processed a given message ignores re-broadcasts of the same ID. When a newer message with the same ID arrives, the existing entry is invalidated and replaced. Convergence: All nodes converge to the same state within seconds with eventual consistency guarantees.

Node Identity and Region

Each node in the cluster has two pieces of identity metadata:

node_id - configured via cluster_config.node_id and surfaced in cluster status output, the React Flow topology view, and diagnostics. The actual memberlist node name is derived from this value combined with the gossip port. If you omit node_id, set one explicitly per pod or instance to make cluster status readable; UUID-style IDs are fine.
region - free-form region label (e.g. "us-east-1", "eu-west") read from cluster_config.region and propagated in node metadata. Defaults to "unknown" when omitted. Region is used for regional leader election and for region-aware operations; it does not gate gossip scope or membership.

Leader Election

Bifrost runs two leader elections in parallel: one cluster-wide and one per region.

Election	Scope	What it does
Cluster leader	All `StateAlive` nodes in the cluster	Coordinates cluster-wide singleton tasks (e.g. pricing URL fetch and broadcast) so only one node hits the upstream and other nodes receive the result via gRPC
Regional leader	Nodes within the same `region` value	Coordinates region-scoped operations

Election is deterministic: the lexicographically-first healthy member wins. The election loop re-evaluates membership every 30 seconds, so leadership transfers automatically when nodes join, leave, or fail. There is nothing to configure - leader election runs whenever clustering is enabled.

Minimum Node Requirements

Recommended: 3+ nodes minimum for optimal fault tolerance.

Cluster Size	Fault Tolerance	Use Case
3 nodes	1 node failure	Small production deployments
5 nodes	2 node failures	Medium production deployments
7+ nodes	3+ node failures	Large enterprise deployments

Configuration Basics

Core Configuration Structure

The new clustering configuration uses a cluster_config object with integrated service discovery:

{
  "cluster_config": {
    "enabled": true,
    "node_id": "bifrost-1",
    "region": "us-east-1",
    "discovery": {
      "enabled": true,
      "type": "kubernetes",
      "service_name": "bifrost-cluster"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    },
    "grpc": {
      "port": 10102,
      "dial_timeout_seconds": 5
    }
  }
}

Required ports for v1.4.x and later:

10101/TCP and 10101/UDP for memberlist gossip (membership and liveness)
10102/TCP for the gRPC counter sync transport (application messages)

Both ports must be reachable peer-to-peer between cluster nodes. NetworkPolicies, security groups, and firewall rules need to allow traffic on both.

Discovery-specific fields (e.g. k8s_label_selector, consul_address, etcd_endpoints) slot into the discovery object alongside type - see each method’s section below.

At startup, cluster mode requires either a non-empty peers list or discovery.enabled: true.

Common Discovery Configuration Fields

All discovery methods support these common fields:

Field	Type	Required	Description
`enabled`	boolean	No	Enable/disable discovery (must be `true` to use discovery at runtime)
`type`	string	Yes	Discovery type: `kubernetes`, `consul`, `etcd`, `dns`, `udp`, `mdns`
`service_name`	string	Conditional	Required for `consul`, `etcd`, `udp`, and typically `mdns`; optional for `kubernetes` and `dns`
`bind_port`	integer	No	Port for cluster communication (default: 10101)
`dial_timeout`	duration	No	Discovery timeout (default: 10s)
`allowed_address_space`	array	No	CIDR ranges to filter discovered nodes (e.g., `["10.0.0.0/8"]`)

Gossip Configuration

Field	Description	Default
`port`	Memberlist gossip port (used for both TCP and UDP)	10101
`timeout_seconds`	Health check timeout	10
`success_threshold`	Successful checks to mark healthy	3
`failure_threshold`	Failed checks to mark unhealthy	3

gRPC Configuration

The gRPC transport carries all application messages and counter sync between nodes. It is enabled automatically whenever cluster_config.enabled is true; configuration is optional.

Field	Description	Default
`port`	TCP port for the cluster gRPC server	10102
`dial_timeout_seconds`	Timeout when dialing a peer for gRPC	5

If you omit the grpc block entirely, both defaults apply. Override only when the defaults conflict with your environment (e.g. another service already binding 10102).

Top-level Fields

Field	Type	Required	Description
`enabled`	boolean	Yes	Master switch for cluster mode
`node_id`	string	Recommended	Logical identifier for this node, used in cluster status and topology views
`region`	string	No	Region label (e.g. `us-east-1`); defaults to `unknown` and gates regional leader election
`peers`	array	Conditional	Static peer list (`host:port` per entry). Required if `discovery.enabled` is `false`
`gossip`	object	No	Memberlist gossip settings (see above)
`grpc`	object	No	gRPC counter sync settings (see above)
`discovery`	object	Conditional	Auto-discovery settings. Required if `peers` is empty

Service Discovery Methods

Bifrost supports 6 service discovery methods to fit any infrastructure. Choose based on your deployment environment:

Kubernetes

Native K8s pod discovery via label selectors

Consul

HashiCorp Consul service mesh integration

etcd

etcd-based distributed discovery

DNS

Traditional DNS SRV record discovery

UDP Broadcast

Local network broadcast discovery

mDNS

Multicast DNS for local development

Kubernetes Discovery

Best for: Kubernetes deployments with StatefulSets or Deployments Kubernetes discovery uses the K8s API to automatically discover pods based on label selectors. This is the most common method for cloud-native deployments.

How It Works

Each Bifrost pod queries the Kubernetes API for pods matching the label selector
Discovers pod IPs automatically as pods scale up/down
Works seamlessly with StatefulSets, Deployments, and DaemonSets
No external dependencies required

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "kubernetes",
      "service_name": "bifrost-cluster",
      "k8s_namespace": "default",
      "k8s_label_selector": "app=bifrost"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`k8s_namespace`	No	Kubernetes namespace to search	`"default"`, `"production"`
`k8s_label_selector`	Yes	Label selector for pod discovery	`"app=bifrost"`, `"app=bifrost,env=prod"`

Kubernetes Deployment Example

StatefulSet
Deployment

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: bifrost
  namespace: default
spec:
  serviceName: bifrost-cluster
  replicas: 3
  selector:
    matchLabels:
      app: bifrost
  template:
    metadata:
      labels:
        app: bifrost
    spec:
      serviceAccountName: bifrost
      containers:
      - name: bifrost
        image: <enterprise_repo_base_url>/bifrost:latest
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 10101
          name: gossip
        - containerPort: 10102
          name: grpc
        volumeMounts:
        - name: config
          mountPath: /etc/bifrost
      volumes:
      - name: config
        configMap:
          name: bifrost-config
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bifrost
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: bifrost-pod-reader
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bifrost-pod-reader
  namespace: default
subjects:
- kind: ServiceAccount
  name: bifrost
  namespace: default
roleRef:
  kind: Role
  name: bifrost-pod-reader
  apiGroup: rbac.authorization.k8s.io

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bifrost
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: bifrost
  template:
    metadata:
      labels:
        app: bifrost
    spec:
      serviceAccountName: bifrost
      containers:
      - name: bifrost
        image: <enterprise_repo_base_url>/bifrost:latest
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 10101
          name: gossip
        - containerPort: 10102
          name: grpc
        volumeMounts:
        - name: config
          mountPath: /etc/bifrost
      volumes:
      - name: config
        configMap:
          name: bifrost-config
---
apiVersion: v1
kind: Service
metadata:
  name: bifrost-cluster
  namespace: default
spec:
  clusterIP: None
  selector:
    app: bifrost
  ports:
  - port: 10101
    name: gossip
  - port: 10102
    name: grpc

Troubleshooting

Pods not discovering each other

Symptoms: Cluster shows only 1 member, pods running in isolationSolutions:

Verify ServiceAccount has RBAC permissions to list pods
Check label selector matches pod labels exactly
Ensure namespace is correct (defaults to “default”)
Verify gossip port (10101) and gRPC port (10102) are not blocked by NetworkPolicies
Check logs for “error listing pods” messages

Permission denied errors

Symptoms: “error getting kubernetes config” or “forbidden” errorsSolutions:

Create ServiceAccount for Bifrost pods
Create Role with get, list, watch permissions on pods
Create RoleBinding linking ServiceAccount to Role
Verify RBAC is enabled in cluster

Cluster forms but nodes show as unhealthy

Symptoms: Nodes discovered but marked as “suspect” or “dead”Solutions:

Verify gossip port (10101) and gRPC port (10102) are accessible between pods
Check for NetworkPolicies blocking pod-to-pod communication
Increase timeout_seconds in gossip config if network is slow
Verify pods are in Running state with kubectl get pods

Consul Discovery

Best for: Consul service mesh environments, multi-datacenter deployments Consul discovery integrates with HashiCorp Consul for service registration and discovery. Ideal for environments already using Consul for service mesh or service discovery.

How It Works

Each Bifrost node registers itself with Consul on startup
Nodes query Consul to discover other Bifrost instances
Consul performs health checks on each node
Unhealthy nodes are automatically deregistered
Supports multi-datacenter deployments

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "consul",
      "service_name": "bifrost-cluster",
      "consul_address": "consul.service.consul:8500"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`consul_address`	No	Consul agent address	`"localhost:8500"`, `"consul.service.consul:8500"` (default: `localhost:8500`)

Consul discovery automatically registers each node with health checks. The health check monitors the gossip port (TCP).

Docker Compose with Consul

version: '3.8'

services:
  consul:
    image: hashicorp/consul:latest
    command: agent -dev -client=0.0.0.0
    ports:
      - "8500:8500"
    networks:
      - bifrost-net

  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config-node1.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
    depends_on:
      - consul
    networks:
      - bifrost-net

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config-node2.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
    depends_on:
      - consul
    networks:
      - bifrost-net

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config-node3.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
    depends_on:
      - consul
    networks:
      - bifrost-net

networks:
  bifrost-net:
    driver: bridge

Troubleshooting

Failed to register with Consul

Symptoms: “failed to register service with Consul” errorsSolutions:

Verify Consul agent is accessible at configured address
Check Consul agent logs for registration errors
Ensure Consul ACL token has write permissions if ACLs enabled
Verify network connectivity between Bifrost and Consul
Check firewall rules allow connections to port 8500

Services registered but not discovered

Symptoms: Consul UI shows services but nodes don’t join clusterSolutions:

Verify service_name matches across all nodes
Check Consul service health checks are passing
Ensure gossip port is accessible between nodes
Verify nodes are registered in correct datacenter
Check for DNS resolution issues if using service DNS names

Health checks failing

Symptoms: Services show as critical in Consul UISolutions:

Verify gossip port (10101) and gRPC port (10102) are accessible
Check Consul agent can reach node’s gossip port
Increase health check timeout in Consul if needed
Review Bifrost logs for startup errors
Ensure nodes have correct IP addresses registered

etcd Discovery

Best for: etcd-based distributed systems, existing etcd infrastructure etcd discovery uses etcd’s distributed key-value store for service registration and discovery. Perfect for environments already using etcd or requiring strong consistency.

How It Works

Each Bifrost node registers itself in etcd with a lease
Nodes maintain lease through keepalive messages
Nodes query etcd prefix to discover other instances
Failed nodes’ leases expire and are automatically removed
Provides strongly consistent service registry

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "etcd",
      "service_name": "bifrost-cluster",
      "etcd_endpoints": [
        "http://etcd-1:2379",
        "http://etcd-2:2379",
        "http://etcd-3:2379"
      ],
      "dial_timeout": "10s"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`etcd_endpoints`	Yes	Array of etcd endpoint URLs	`["http://localhost:2379"]`, `["https://etcd1:2379", "https://etcd2:2379"]`
`dial_timeout`	No	Connection timeout	`"10s"` (default), `"30s"`

Each node registers under /services/{service_name}/{node_id} with a 30-second TTL lease.

Docker Compose with etcd

version: '3.8'

services:
  etcd:
    image: quay.io/coreos/etcd:latest
    command:
      - etcd
      - --advertise-client-urls=http://etcd:2379
      - --listen-client-urls=http://0.0.0.0:2379
      - --listen-peer-urls=http://0.0.0.0:2380
      - --initial-cluster=etcd=http://etcd:2380
      - --initial-advertise-peer-urls=http://etcd:2380
    ports:
      - "2379:2379"
      - "2380:2380"
    networks:
      - bifrost-net

  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
    depends_on:
      - etcd
    networks:
      - bifrost-net

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
    depends_on:
      - etcd
    networks:
      - bifrost-net

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
    depends_on:
      - etcd
    networks:
      - bifrost-net

networks:
  bifrost-net:
    driver: bridge

Troubleshooting

Failed to create etcd client

Symptoms: “etcd client error” on startupSolutions:

Verify etcd endpoints are accessible
Check URL format (http:// or https://)
Ensure etcd cluster is healthy and running
Verify network connectivity to etcd endpoints
Check firewall rules allow connections to port 2379
Increase dial_timeout if network is slow

Failed to register with etcd

Symptoms: “failed to register with etcd” errorsSolutions:

Verify etcd cluster is accepting writes
Check etcd cluster has available space
Ensure authentication credentials if etcd has auth enabled
Review etcd logs for permission or quota errors
Verify node can resolve etcd hostnames

Lease keepalive failures

Symptoms: Nodes repeatedly registering/deregisteringSolutions:

Check network stability between nodes and etcd
Verify etcd cluster is not overloaded
Monitor etcd metrics for high latency
Increase lease TTL if network has high latency
Check for etcd leader election issues

DNS Discovery

Best for: Traditional infrastructure, static node addresses, cloud DNS services DNS discovery uses standard DNS resolution to discover cluster nodes. Works with any DNS server and is ideal for static deployments or cloud environments with DNS integration.

How It Works

Configure DNS A records or SRV records for cluster nodes
Bifrost queries DNS to resolve configured names
All returned IP addresses are treated as potential cluster members
Supports multiple DNS names for different node groups
Works with internal DNS, cloud DNS, or public DNS

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "dns",
      "service_name": "bifrost-cluster",
      "dns_names": [
        "bifrost-cluster.local",
        "bifrost-nodes.internal.company.com"
      ],
      "bind_port": 10101
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`dns_names`	Yes	Array of DNS names to resolve	`["bifrost.local"]`, `["node1.local", "node2.local", "node3.local"]`
`bind_port`	No	Port appended to discovered IPs	`10101` (default)

DNS discovery is passive - it doesn’t register nodes. You must manage DNS records externally (via DNS server, cloud DNS, or Kubernetes DNS).

Setup Examples

Cloud DNS (AWS Route53)
Kubernetes Headless Service
Local DNS (dnsmasq)

# Create A records for each node
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "bifrost-cluster.internal.company.com",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [
          {"Value": "10.0.1.10"},
          {"Value": "10.0.1.11"},
          {"Value": "10.0.1.12"}
        ]
      }
    }]
  }'

apiVersion: v1
kind: Service
metadata:
  name: bifrost-cluster
  namespace: default
spec:
  clusterIP: None  # Headless service
  selector:
    app: bifrost
  ports:
  - port: 10101
    name: gossip
---
# DNS will resolve bifrost-cluster.default.svc.cluster.local
# to all pod IPs matching the selector

# /etc/dnsmasq.conf
address=/bifrost-cluster.local/192.168.1.10
address=/bifrost-cluster.local/192.168.1.11
address=/bifrost-cluster.local/192.168.1.12

# Or use /etc/hosts on each node
echo "192.168.1.10 node1.bifrost.local" >> /etc/hosts
echo "192.168.1.11 node2.bifrost.local" >> /etc/hosts
echo "192.168.1.12 node3.bifrost.local" >> /etc/hosts

Troubleshooting

DNS lookup errors

Symptoms: “dns lookup error” in logs, no nodes discoveredSolutions:

Verify DNS names are resolvable: nslookup bifrost-cluster.local
Check DNS server is accessible from Bifrost nodes
Verify /etc/resolv.conf has correct nameserver
Test DNS resolution from inside container if using Docker
Check for DNS caching issues (try flushing DNS cache)

No nodes discovered via DNS

Symptoms: DNS resolves but cluster has 0 membersSolutions:

Verify DNS returns multiple A records (not CNAME)
Check that returned IPs are correct and reachable
Ensure bind_port matches actual gossip port on nodes
Verify nodes are listening on returned IP addresses
Use dig or nslookup to verify DNS response format

Nodes discovered but can't connect

Symptoms: IPs discovered but gossip connection failsSolutions:

Verify gossip port (10101) and gRPC port (10102) are open on all nodes
Check firewall rules between nodes
Ensure nodes are listening on correct network interface
Verify IP addresses match node’s actual network addresses
Test connectivity: telnet <ip> 10101

UDP Broadcast Discovery

Best for: Local network deployments, on-premise infrastructure, development clusters UDP broadcast discovery automatically finds nodes on the same local network using broadcast packets. No external dependencies required.

How It Works

Nodes broadcast UDP discovery beacons on configured port
Other nodes on the same network respond with acknowledgments
Nodes discover each other’s IP addresses automatically
Limited to nodes on the same broadcast domain (subnet)
Requires allowed_address_space for security

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "udp",
      "service_name": "bifrost-cluster",
      "udp_broadcast_port": 9999,
      "allowed_address_space": [
        "192.168.1.0/24",
        "10.0.0.0/8"
      ],
      "dial_timeout": "10s"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`udp_broadcast_port`	Yes	Port for broadcast discovery	`9999`, `8888`
`allowed_address_space`	Yes	CIDR ranges to limit discovery scope	`["192.168.1.0/24"]`, `["10.0.0.0/8", "172.16.0.0/12"]`
`dial_timeout`	No	Time to wait for responses	`"10s"` (default)

UDP broadcast discovery requires allowed_address_space to be configured. This prevents scanning arbitrary networks and limits discovery to trusted subnets.

Docker Compose Example

version: '3.8'

services:
  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    network_mode: bridge
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
      - "9999:9999/udp"
      - "10101:10101"
      - "10102:10102"

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    network_mode: bridge
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
      - "9999:9999/udp"
      - "10101:10101"
      - "10102:10102"

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    network_mode: bridge
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
      - "9999:9999/udp"
      - "10101:10101"
      - "10102:10102"

Use network_mode: bridge (default) or host for UDP broadcast. Custom networks may not support broadcast.

Troubleshooting

No nodes discovered via UDP broadcast

Symptoms: Discovery runs but finds 0 nodesSolutions:

Verify allowed_address_space includes node IP addresses
Check UDP broadcast port is open (firewall/security groups)
Ensure nodes are on same subnet/broadcast domain
Verify broadcast is enabled on network interface
Test with tcpdump -i any -n udp port 9999
Check Docker network mode supports broadcast (use bridge or host)

Address space filtering issues

Symptoms: “not in allowed address space” warningsSolutions:

Verify CIDR notation is correct (e.g., 192.168.1.0/24)
Ensure allowed_address_space covers all node IPs
Check node IP addresses: ip addr or ifconfig
Remember to use network address, not host address
Test CIDR match online or with ipcalc

Permission denied on UDP port

Symptoms: “permission denied” or “address already in use”Solutions:

Check if another process is using the UDP broadcast port
Verify port number is > 1024 (non-privileged) or run as root
Use netstat -tulpn | grep 9999 to check port usage
Change udp_broadcast_port to different value
Ensure firewall isn’t blocking UDP on that port

mDNS Discovery

Best for: Local development, testing, zero-configuration setups mDNS (Multicast DNS) provides zero-configuration service discovery on local networks. Perfect for development and testing without requiring any infrastructure setup.

How It Works

Nodes advertise themselves via mDNS (Bonjour/Avahi)
Other nodes browse for mDNS services
Automatic discovery within the same local network
No DNS server or configuration required
Limited to local network segment

Configuration

{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "mdns",
      "service_name": "bifrost",
      "mdns_service": "_bifrost._tcp",
      "dial_timeout": "10s"
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  }
}

Configuration Parameters

Parameter	Required	Description	Example
`mdns_service`	No	mDNS service type	`"_bifrost._tcp"` (default), `"_myapp._tcp"`
`dial_timeout`	No	Time to wait for mDNS responses	`"10s"` (default)

mDNS is designed for development and testing. For production, use Kubernetes, Consul, or etcd discovery.

Local Development Example

# Start first node
docker run -p 8080:8080 -p 10101:10101 -p 10201:10102 \
  -v $(pwd)/config-mdns.json:/etc/bifrost/config.json \
  <enterprise_repo_base_url>/bifrost:latest

# Start second node (discovers first automatically)
docker run -p 8081:8080 -p 10111:10101 -p 10211:10102 \
  -v $(pwd)/config-mdns.json:/etc/bifrost/config.json \
  <enterprise_repo_base_url>/bifrost:latest

# Start third node (discovers both automatically)
docker run -p 8082:8080 -p 10121:10101 -p 10221:10102 \
  -v $(pwd)/config-mdns.json:/etc/bifrost/config.json \
  <enterprise_repo_base_url>/bifrost:latest

Troubleshooting

mDNS services not discovered

Symptoms: Nodes don’t discover each other via mDNSSolutions:

Verify mDNS is enabled on network (check firewall)
Ensure multicast is enabled on network interface
Check nodes are on same local network segment
Verify mDNS port 5353 is not blocked
Test mDNS resolution: avahi-browse -a (Linux) or dns-sd -B (macOS)
Increase dial_timeout if discovery is slow

Network address validation errors

Symptoms: “skipping invalid host address” warningsSolutions:

This is normal - mDNS returns network/broadcast addresses
mDNS automatically filters invalid addresses (127.x.x.x, *.0, *.255)
Check that nodes have valid non-loopback IP addresses
Ensure nodes are not using 127.0.0.1 for binding
Verify network interface has proper IP configuration

Discovery works but cluster unstable

Symptoms: Nodes discover then disconnect repeatedlySolutions:

mDNS has eventual consistency, allow time for propagation
Check gossip port accessibility between nodes
Verify network doesn’t drop multicast packets
Consider using a more robust discovery method for production
Check for network congestion or packet loss

Deployment Patterns

Docker Compose Deployment

Complete example using Kubernetes-style discovery with a shared config store:

version: '3.8'

services:
  postgres:
    image: postgres:14
    environment:
      POSTGRES_DB: bifrost
      POSTGRES_USER: bifrost
      POSTGRES_PASSWORD: bifrost_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - bifrost-net

  consul:
    image: hashicorp/consul:latest
    command: agent -dev -client=0.0.0.0
    ports:
      - "8500:8500"
    networks:
      - bifrost-net

  bifrost-1:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8080:8080"
    depends_on:
      - postgres
      - consul
    networks:
      - bifrost-net

  bifrost-2:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8081:8080"
    depends_on:
      - postgres
      - consul
    networks:
      - bifrost-net

  bifrost-3:
    image: <enterprise_repo_base_url>/bifrost:latest
    environment:
      - BIFROST_CONFIG=/etc/bifrost/config.json
    volumes:
      - ./config.json:/etc/bifrost/config.json
    ports:
      - "8082:8080"
    depends_on:
      - postgres
      - consul
    networks:
      - bifrost-net

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - bifrost-1
      - bifrost-2
      - bifrost-3
    networks:
      - bifrost-net

volumes:
  postgres_data:

networks:
  bifrost-net:
    driver: bridge

nginx.conf for load balancing:

events {
    worker_connections 1024;
}

http {
    upstream bifrost_cluster {
        least_conn;
        server bifrost-1:8080 max_fails=3 fail_timeout=30s;
        server bifrost-2:8080 max_fails=3 fail_timeout=30s;
        server bifrost-3:8080 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 80;
        
        location / {
            proxy_pass http://bifrost_cluster;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            
            # Timeouts
            proxy_connect_timeout 60s;
            proxy_send_timeout 60s;
            proxy_read_timeout 60s;
        }
        
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
}

Kubernetes Production Deployment

Production-ready Kubernetes deployment with StatefulSet:

If you use PostgreSQL for config_store, ensure the target database is UTF8 encoded. See PostgreSQL UTF8 Requirement.

apiVersion: v1
kind: ConfigMap
metadata:
  name: bifrost-config
  namespace: bifrost
data:
  config.json: |
    {
      "cluster_config": {
        "enabled": true,
        "discovery": {
          "enabled": true,
          "type": "kubernetes",
          "service_name": "bifrost-cluster",
          "k8s_namespace": "bifrost",
          "k8s_label_selector": "app=bifrost,component=gateway"
        },
        "gossip": {
          "port": 10101,
          "config": {
            "timeout_seconds": 10,
            "success_threshold": 3,
            "failure_threshold": 3
          }
        }
      },
      "config_store": {
        "enabled": true,
        "type": "postgres",
        "config": {
          "host": "postgres.bifrost.svc.cluster.local",
          "port": "5432",
          "user": "bifrost",
          "password": "changeme",
          "db_name": "bifrost",
          "ssl_mode": "require"
        }
      }
    }
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bifrost
  namespace: bifrost
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: bifrost-pod-reader
  namespace: bifrost
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bifrost-pod-reader
  namespace: bifrost
subjects:
- kind: ServiceAccount
  name: bifrost
  namespace: bifrost
roleRef:
  kind: Role
  name: bifrost-pod-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: bifrost
  namespace: bifrost
spec:
  serviceName: bifrost-cluster
  replicas: 3
  selector:
    matchLabels:
      app: bifrost
      component: gateway
  template:
    metadata:
      labels:
        app: bifrost
        component: gateway
    spec:
      serviceAccountName: bifrost
      containers:
      - name: bifrost
        image: <enterprise_repo_base_url>/bifrost:latest
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 10101
          name: gossip
          protocol: TCP
        - containerPort: 10102
          name: grpc
          protocol: TCP
        env:
        - name: BIFROST_CONFIG
          value: /etc/bifrost/config.json
        volumeMounts:
        - name: config
          mountPath: /etc/bifrost
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: config
        configMap:
          name: bifrost-config
---
apiVersion: v1
kind: Service
metadata:
  name: bifrost-cluster
  namespace: bifrost
spec:
  clusterIP: None
  selector:
    app: bifrost
    component: gateway
  ports:
  - port: 10101
    name: gossip
    protocol: TCP
  - port: 10102
    name: grpc
    protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: bifrost
  namespace: bifrost
spec:
  type: LoadBalancer
  selector:
    app: bifrost
    component: gateway
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: bifrost-pdb
  namespace: bifrost
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: bifrost
      component: gateway

Bare Metal / VM Deployment

For bare metal or VM deployments using systemd: Step 1: Install Bifrost on each node

# Download Bifrost Enterprise binary
curl -O https://releases.getmaxim.ai/bifrost-enterprise/latest/bifrost-enterprise-linux-amd64
chmod +x bifrost-enterprise-linux-amd64
sudo mv bifrost-enterprise-linux-amd64 /usr/local/bin/bifrost-enterprise

Step 2: Create configuration file

sudo mkdir -p /etc/bifrost
sudo cat > /etc/bifrost/config.json <<EOF
{
  "cluster_config": {
    "enabled": true,
    "discovery": {
      "enabled": true,
      "type": "dns",
      "service_name": "bifrost-cluster",
      "dns_names": ["bifrost-cluster.internal.company.com"]
    },
    "gossip": {
      "port": 10101,
      "config": {
        "timeout_seconds": 10,
        "success_threshold": 3,
        "failure_threshold": 3
      }
    }
  },
  "config_store": {
    "enabled": true,
    "type": "postgres",
    "config": {
      "host": "postgres.internal.company.com",
      "port": "5432",
      "user": "bifrost",
      "password": "secure_password",
      "db_name": "bifrost",
      "ssl_mode": "require"
    }
  }
}
EOF

Step 3: Create systemd service

sudo cat > /etc/systemd/system/bifrost.service <<EOF
[Unit]
Description=Bifrost Enterprise API Gateway
After=network.target

[Service]
Type=simple
User=bifrost
Group=bifrost
Environment="BIFROST_CONFIG=/etc/bifrost/config.json"
ExecStart=/usr/local/bin/bifrost-enterprise
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal

# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/bifrost

[Install]
WantedBy=multi-user.target
EOF

Step 4: Setup DNS records

# Add A records for bifrost-cluster.internal.company.com
# pointing to all node IPs:
# 10.0.1.10  (node1)
# 10.0.1.11  (node2)
# 10.0.1.12  (node3)

Step 5: Start and enable service

sudo useradd -r -s /bin/false bifrost
sudo mkdir -p /var/lib/bifrost
sudo chown bifrost:bifrost /var/lib/bifrost
sudo systemctl daemon-reload
sudo systemctl enable bifrost
sudo systemctl start bifrost
sudo systemctl status bifrost

Step 6: Verify cluster formation

# Check logs on each node
sudo journalctl -u bifrost -f

# Look for messages like:
# "successfully joined X peers on startup"
# "cluster health: HEALTHY"

Cluster Operations

Cluster Topology View

The admin UI ships an interactive cluster graph (React Flow) that renders the live cluster as a circle of nodes with edges colored by reachability. Each node card shows its node_id, region, current state (alive / suspect / dead / left), and a leader badge if it currently holds either the cluster-wide or regional leadership. The view updates in the background and triggers an automatic diagnostic on leader transitions. Single-node clusters render the simplified single-card view.

Cluster Diagnostics

A diagnostic flow lets administrators verify in-cluster reachability end-to-end. When triggered, the local node broadcasts a no-op diagnostic ping to every peer that advertises the ack:v1 capability and streams ACK status back to the caller as each peer responds. Use it to:

Confirm that a newly added node is receiving and acknowledging cluster messages
Surface peers that are partitioned or silently dropping traffic
Validate that NetworkPolicies / firewall rules permit gRPC after a config change

The diagnostic does not mutate state on any node - the message is purely a round-trip probe. It is admin-only and is invoked from the cluster topology view or the equivalent admin endpoint.

Mixed-Version Rollouts

Bifrost negotiates per-peer capabilities so newer and older cluster members can run side-by-side during rolling upgrades. Peers advertise the ack:v1 capability in their gossip metadata; a node only tracks ACKs from peers that advertise it. Older peers still receive broadcasts but are excluded from the pending-ACK set so they don’t trigger false “unacked” metrics or unnecessary retries. In practice this means: you can roll out a new Bifrost version one pod at a time, and the cluster will degrade ACK tracking gracefully for the older pods until they’re replaced. Rolling-update strategies (Kubernetes RollingUpdate, Nomad canaries, etc.) are supported without quorum loss.

Leader-Coordinated Tasks

A small number of cluster-wide tasks run only on the elected leader, with results broadcast over gRPC so followers stay in sync without each one independently doing the work. The clearest example is pricing sync: only the leader fetches the upstream pricing URL on the configured interval and then broadcasts a database reload message; followers reload from the local store rather than each making the upstream HTTP call. Region-scoped variants of this pattern run on the regional leader. There is no configuration to enable this - leader-coordinated tasks engage automatically when clustering is enabled. If the leader fails, the next election (within ~30 seconds) hands the responsibility to a new node.

Replicated Entity Types

Every replicated message carries an EntityType identifying the kind of state being broadcast. The cluster replicates the following entity types over gRPC:

Category	Entity Types
Catalog & Providers	`model_catalog`, `provider`, `model_config`, `pricing`, `pricing_override`
Governance	`governance`, `virtual_key`, `team`, `customer`, `business_unit_governance`, `user_governance`, `access_profile`, `api_key`
Security & Auth	`auth_config`, `rbac`, `proxy_config`, `client_config`
Routing & Load Balancing	`routing_rule`, `load_balancing`, `load_balancing_log`
MCP	`mcp_tool`, `mcp_tool_group`
Guardrails	`guardrail`, `guardrail_config`, `guardrail_sampling`
Plugins & Connectors	`plugin`, `observability_connector`
Prompts & Storage	`prompt_deployment`, `kv_store`
Cluster	`cluster_diagnostic` (no-op probe used by the diagnostic flow)

Each entity type follows the same broadcast and dedup rules: a unique message ID, a SentAt timestamp, and a deduper TTL on the receiver side. Newer messages with the same ID invalidate older ones, so a late-arriving stale message will not overwrite fresh state.

Troubleshooting

General Clustering Issues

Cluster forms but only has 1 member

Symptoms: Each node thinks it’s the only memberCommon Causes & Solutions:

Discovery not configured: Verify discovery.enabled: true and discovery.type is set
Service name mismatch: Ensure all nodes have identical service_name
Gossip port blocked: Check firewall allows TCP port 10101 between nodes
Discovery method issues: See method-specific troubleshooting above
Network isolation: Verify nodes can reach each other on gossip port

Split brain - nodes form separate clusters

Symptoms: Nodes divided into separate clustersCommon Causes & Solutions:

Network partition: Check network connectivity between all nodes
Different discovery configs: Ensure all nodes use same discovery settings
Firewall blocking gossip: Verify bidirectional connectivity on port 10101
Discovery scoped incorrectly: Check label selectors, DNS names, or address spaces
Restart all nodes: Sometimes requires simultaneous restart to reform cluster

High memory usage in cluster

Symptoms: Memory grows over time, especially in large clustersCommon Causes & Solutions:

Large gossip messages: Check size of gossiped data
Too many nodes: Optimize for clusters with 3-7 nodes typically
Message deduplication cache: This is normal, cache TTL is 2 minutes
Increase node resources: Ensure adequate memory allocation

Cluster unstable - nodes flapping

Symptoms: Nodes repeatedly join and leave clusterCommon Causes & Solutions:

Network instability: Check for packet loss or high latency
Resource constraints: Ensure nodes have adequate CPU/memory
Timeout too aggressive: Increase timeout_seconds in gossip config
Health check failures: Review liveness probe configuration
Discovery intervals: Check discovery isn’t running too frequently

Cannot broadcast messages to cluster

Symptoms: Broadcast queue errors, messages not propagatingCommon Causes & Solutions:

Queue not initialized: Check logs for initialization errors
No active members: Verify cluster has multiple healthy members
Gossip port unreachable: Test connectivity between all nodes
Message too large: Check size of broadcast messages

Key log messages to look for:

✅ Successful cluster formation:
- "successfully joined X peers on startup"
- "cluster health: HEALTHY"
- "discovered X nodes"

⚠️ Warning signs:
- "no new nodes discovered"
- "failed to join cluster"
- "cluster health: NOT HEALTHY"
- "node marked as suspect"

❌ Errors:
- "discovery failed"
- "failed to broadcast"
- "timeout waiting for response"

Health Check Endpoints

Monitor cluster health via HTTP endpoints:

# Check if node is healthy
curl http://localhost:8080/health

# Get cluster status (if exposed)
curl http://localhost:8080/cluster/status

# Expected response shows all cluster members
{
  "local_node": "bifrost-remote-10101-...",
  "members": 3,
  "healthy_members": 3,
  "cluster_health": "HEALTHY"
}

This clustering implementation ensures Bifrost can handle enterprise-scale deployments with high availability, automatic service discovery, and intelligent traffic distribution across any infrastructure.

Overview

Migration Guides

Features

Governance & Access Control

Security

Documentation Index

​Overview

​Why Clustering Matters

​Core Features

​Architecture

​Peer-to-Peer Network Design

​Cluster Communication

​Application messages and entity types

​Message dedup and invalidation

​Node Identity and Region

​Leader Election

​Minimum Node Requirements

​Configuration Basics

​Core Configuration Structure

​Common Discovery Configuration Fields

​Gossip Configuration

​gRPC Configuration

​Top-level Fields

​Service Discovery Methods

Kubernetes

Consul

etcd

DNS

UDP Broadcast

mDNS

​Kubernetes Discovery

​How It Works

​Configuration

​Configuration Parameters

​Kubernetes Deployment Example

​Troubleshooting

​Consul Discovery

​How It Works

​Configuration

​Configuration Parameters

​Docker Compose with Consul

​Troubleshooting

​etcd Discovery

​How It Works

​Configuration

​Configuration Parameters

​Docker Compose with etcd

​Troubleshooting

​DNS Discovery

​How It Works

​Configuration

​Configuration Parameters

​Setup Examples

​Troubleshooting

​UDP Broadcast Discovery

​How It Works

​Configuration

​Configuration Parameters

​Docker Compose Example

​Troubleshooting

​mDNS Discovery

​How It Works

​Configuration

​Configuration Parameters

​Local Development Example

​Troubleshooting

​Deployment Patterns

​Docker Compose Deployment

​Kubernetes Production Deployment

​Bare Metal / VM Deployment

​Cluster Operations

​Cluster Topology View

​Cluster Diagnostics

​Mixed-Version Rollouts

​Leader-Coordinated Tasks

​Replicated Entity Types

​Troubleshooting

​General Clustering Issues

​Health Check Endpoints

Overview

Why Clustering Matters

Core Features

Architecture

Peer-to-Peer Network Design

Cluster Communication

Application messages and entity types

Message dedup and invalidation

Node Identity and Region

Leader Election

Minimum Node Requirements

Configuration Basics

Core Configuration Structure

Common Discovery Configuration Fields

Gossip Configuration

gRPC Configuration

Top-level Fields

Service Discovery Methods

Kubernetes Discovery

How It Works

Configuration

Configuration Parameters

Kubernetes Deployment Example

Troubleshooting

Consul Discovery

How It Works

Configuration

Configuration Parameters

Docker Compose with Consul

Troubleshooting

etcd Discovery

How It Works

Configuration

Configuration Parameters

Docker Compose with etcd

Troubleshooting

DNS Discovery

How It Works

Configuration

Configuration Parameters

Setup Examples

Troubleshooting

UDP Broadcast Discovery

How It Works

Configuration

Configuration Parameters

Docker Compose Example

Troubleshooting

mDNS Discovery

How It Works

Configuration

Configuration Parameters

Local Development Example

Troubleshooting

Deployment Patterns

Docker Compose Deployment

Kubernetes Production Deployment

Bare Metal / VM Deployment

Cluster Operations

Cluster Topology View

Cluster Diagnostics

Mixed-Version Rollouts

Leader-Coordinated Tasks

Replicated Entity Types

Troubleshooting

General Clustering Issues

Health Check Endpoints