Datadog

​

Overview

• APM Traces - Distributed tracing via dd-trace-go v2 with W3C Trace Context support for end-to-end request visibility
• LLM Observability - Native Datadog LLM Obs integration for AI/ML-specific monitoring
• Metrics - Operational metrics via DogStatsD or the Metrics API

​

Deployment Modes

​

Agent Mode

• APM Traces → Agent at localhost:8126
• Metrics → DogStatsD at localhost:8125

​

Agentless Mode

• LLM Observability → https://api.{site}/api/intake/llm-obs/v1/trace/spans
• Metrics → https://api.{site} Metrics API (series to /api/v2/series, distributions to /api/v1/distribution_points)

​

Configuration

​

Required Fields

​

Environment Variable Substitution

{
  "api_key": "env.DD_API_KEY",
  "agent_addr": "env.DD_AGENT_ADDR",
  "dogstatsd_addr": "env.DD_DOGSTATSD_ADDR",
  "custom_tags": {
    "team": "env.TEAM_NAME",
    "cost_center": "env.COST_CENTER"
  }
}

​

Separate host and port (Kubernetes)

{
  "dogstatsd_host": "env.DD_AGENT_HOST",
  "agent_host": "env.DD_AGENT_HOST"
}

env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

​

Setup

• UI
• Go SDK
• config.json

Configure the Datadog plugin through the Bifrost UI:

1. Navigate to Plugins
2. Enable the Datadog plugin
3. Configure the required fields based on your deployment mode

package main

import (
    "context"
    bifrost "github.com/maximhq/bifrost/core"
    "github.com/maximhq/bifrost/core/schemas"
    "github.com/maximhq/bifrost/framework/modelcatalog"
    datadog "github.com/maximhq/bifrost-enterprise/plugins/datadog"
)

func main() {
    ctx := context.Background()
    logger := schemas.NewLogger()
    
    // Initialize model catalog (required for cost calculation)
    modelCatalog := modelcatalog.NewModelCatalog(logger)
    
    // Agent mode configuration
    ddPlugin, err := datadog.Init(ctx, &datadog.Config{
        ServiceName: "my-llm-service",
        Env:         "production",
        Version:     "1.0.0",
        CustomTags: map[string]string{
            "team": "platform",
        },
    }, logger, modelCatalog, "1.0.0")
    if err != nil {
        panic(err)
    }
    
    // Initialize Bifrost with the plugin
    client, err := bifrost.Init(ctx, schemas.BifrostConfig{
        Account: &yourAccount,
        Plugins: []schemas.Plugin{ddPlugin},
    })
    if err != nil {
        panic(err)
    }
    defer client.Shutdown()
    
    // All requests are now traced to Datadog
}

For agentless mode:

// Agentless mode configuration
enableAgentless := true
ddPlugin, err := datadog.Init(ctx, &datadog.Config{
    ServiceName: "my-llm-service",
    Env:         "production",
    Agentless:   &enableAgentless,
    APIKey:      &schemas.EnvVar{EnvVarName: "DD_API_KEY"},
    Site:        "datadoghq.com",
}, logger, modelCatalog, "1.0.0")

​

Agent Mode (Minimal)

{
  "plugins": [
    {
      "enabled": true,
      "name": "datadog",
      "config": {
        "service_name": "bifrost",
        "env": "production"
      }
    }
  ]
}

​

Agent Mode (Full Configuration)

{
  "plugins": [
    {
      "enabled": true,
      "name": "datadog",
      "config": {
        "service_name": "my-llm-gateway",
        "ml_app": "my-ml-application",
        "agent_addr": "localhost:8126",
        "dogstatsd_addr": "localhost:8125",
        "env": "production",
        "version": "1.2.3",
        "custom_tags": {
          "team": "platform",
          "cost_center": "env.COST_CENTER"
        },
        "enable_metrics": true,
        "enable_traces": true,
        "enable_llm_obs": true
      }
    }
  ]
}

​

Agentless Mode

{
  "plugins": [
    {
      "enabled": true,
      "name": "datadog",
      "config": {
        "service_name": "my-llm-gateway",
        "env": "production",
        "agentless": true,
        "api_key": "env.DD_API_KEY",
        "site": "datadoghq.com"
      }
    }
  ]
}

Set the environment variable:

export DD_API_KEY="your-datadog-api-key"

​

Datadog Sites

​

LLM Observability

​

ML App Grouping

{
  "service_name": "bifrost-gateway",
  "ml_app": "customer-support-ai"
}

• Group related LLM operations across multiple services
• Track costs and performance by application
• Apply ML-specific alerts and dashboards

​

Session Tracking

curl -X POST https://your-bifrost-gateway/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "x-bf-session-id: user-123-session-456" \
  -d '{...}'

​

Grouping APM Traces by Session

{
  "group_traces_by_session": true
}

# Both requests below land in the same APM trace
curl ... -H "x-bf-session-id: user-123-session-456" -d '{...}'
curl ... -H "x-bf-session-id: user-123-session-456" -d '{...}'

​

W3C Distributed Tracing

curl -X POST https://your-bifrost-gateway/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01" \
  -d '{...}'

• End-to-end visibility - See LLM calls in the context of your full application trace
• Cross-service correlation - Link frontend requests → backend services → Bifrost → LLM providers
• Latency attribution - Understand how LLM latency contributes to overall request time

traceparent: {version}-{trace-id}-{parent-id}-{trace-flags}

​

What’s Captured

• Input/Output Messages - Full conversation history with role attribution
• Token Usage - Input, output, and total token counts
• Cost - Calculated cost in USD based on model pricing
• Latency - Request duration and time-to-first-token for streaming
• Model Info - Provider, model name, and request parameters
• Tool Calls - Function/tool call details for agentic workflows

​

Metrics Reference

​

Migrating from bifrost.cost.usd

1. Replace bifrost.cost.usd with bifrost.request.cost.usd in all queries.
2. Update aggregations for Distribution semantics — each sample is one request’s cost:
   • Total spend: sum:bifrost.request.cost.usd{*} (do not append .as_count() or .rollup(sum); the sum: aggregator already returns the additive total)
   • Per-request statistics: avg:, max:, or percentile aggregators
3. Recreate monitors and alerts on the new metric, adjusting thresholds if they assumed gauge behavior (the gauge systematically under-reported under concurrent load).

​

Custom Tags

• provider - LLM provider (openai, anthropic, etc.)
• model - Model name
• method - Type of request (chat, embedding, etc.)
• bifrost_node - Per-instance identity (BIFROST_NODE_ID if set, otherwise hostname-pid)
• plus Bifrost-context tags when available (virtual key, selected key, team, customer, fallback index)

​

Captured Data

​

Span Attributes

• Span Name - Based on request type (genai.chat, genai.embedding, etc.)
• Service Info - service.name, service.version, env (Datadog’s unified service tagging: service, version, env)
• Provider & Model - gen_ai.provider.name, gen_ai.request.model

​

Request Parameters

• Temperature, max_tokens, top_p, stop sequences
• Presence/frequency penalties
• Tool configurations and parallel tool calls
• Custom parameters via ExtraParams

​

Input/Output Data

• Complete chat history with role-based messages
• Prompt text for completions
• Response content with role attribution
• Tool calls and results
• Reasoning and refusal content (when present)

​

Performance Metrics

• Token usage (prompt, completion, total)
• Cost calculations in USD
• Latency and timing (start/end timestamps)
• Time to first token (streaming)
• Error details with status codes

​

Bifrost Context

• Virtual key ID and name
• Selected key ID and name
• Team ID and name
• Customer ID and name
• Retry count and fallback index

​

Plugin Span Filtering

{
  "plugins": [
    {
      "name": "datadog",
      "enabled": true,
      "config": {
        "service_name": "bifrost",
        "agent_addr": "localhost:8126",
        "enable_traces": true,
        "plugin_span_filter": {
          "mode": "exclude",
          "plugins": ["logging", "compat", "telemetry"]
        }
      }
    }
  ]
}

​

Supported Request Types

​

When to Use

​

Datadog Plugin

• Use Datadog as your primary observability platform
• Want native LLM Observability integration with ML App grouping
• Need seamless correlation with existing Datadog APM traces via W3C distributed tracing
• Require Datadog-specific features like notebooks and dashboards
• Want session tracking for conversation flows

​

vs. OTel Plugin

• Need multi-vendor observability (send to multiple backends)
• Are using Datadog via an OpenTelemetry Collector
• Want vendor flexibility to switch backends without code changes
• Prefer standardized OpenTelemetry semantic conventions

​

vs. Built-in Observability

• Local development and testing
• Simple self-hosted deployments
• No external dependencies required
• Direct database access to logs

​

Troubleshooting

​

Agent Connectivity Issues

# Check agent status
datadog-agent status

# Test APM endpoint
curl -v http://localhost:8126/info

# Test DogStatsD (should accept UDP packets)
echo "test.metric:1|c" | nc -u -w1 localhost 8125

​

Agentless Mode Not Working

1. Verify your API key is valid:

curl -X GET "https://api.datadoghq.com/api/v1/validate" \
  -H "DD-API-KEY: $DD_API_KEY"

2. Ensure the site matches your API key’s region
3. Check that the API key environment variable is set:

echo $DD_API_KEY

​

Missing Traces

1. Enable debug logging in Bifrost:

bifrost-http --log-level debug

2. Verify traces are enabled in your configuration:

{
  "enable_traces": true,
  "enable_llm_obs": true
}

3. Check for errors in the Bifrost logs related to the Datadog plugin

​

Missing Metrics

1. Verify DogStatsD is running (agent mode):

datadog-agent status | grep DogStatsD

2. Ensure metrics are enabled:

{
  "enable_metrics": true
}

3. For agentless mode, verify your API key has metrics submission permissions

​

LLM Observability Not Appearing

1. LLM Observability requires enable_llm_obs: true (default)
2. Verify your Datadog plan includes LLM Observability
3. Check the ML App name in Datadog under LLM Observability → Applications

​

Next Steps

• OTel Plugin - OpenTelemetry integration for multi-vendor observability
• Built-in Observability - Local logging for development
• Telemetry - Prometheus metrics and dashboards