Log Exports

Bifrost’s log store can be paired with an object storage backend (S3, GCS, or Azure Blob Storage) so that large request/response payloads are streamed to durable object storage while the logs database (SQLite, Postgres, or ClickHouse) keeps only searchable metadata, indexes, and pointers. This keeps the database small and fast, makes payloads cheap to retain for long periods, and lets you query archived traffic from your own data lake.

How it works

object_storage is not a standalone feature. It is a sub-config of logs_store:

logs_store
├── enabled
├── type            (sqlite | postgres | clickhouse)
├── config          (SQLite path, or Postgres / ClickHouse connection)
├── object_storage              ← S3, GCS, or Azure payload offload (optional)
├── object_storage_exclude_fields
└── object_storage_exclude_request_types

Retention is configured separately, at client_config.log_retention_days - see Retention policy below. When logs_store.object_storage is set:

Bifrost writes per-request metadata (timestamps, provider, model, latency, token counts, cost, status, IDs) to the logs database, along with each failed request’s error details (except on content-hidden logs, see below), so error type, error code and status code breakdowns keep working.
Large payload fields (request body, response body, streamed chunks, tool call arguments, etc.) are uploaded to the configured bucket under prefix/.
The database row stores the object key, so the UI and API can fetch the payload on demand.
object_storage_exclude_fields lets you skip specific fields from offload (for example, when you do not want to retain raw user prompts at all).
object_storage_exclude_request_types lets you skip whole request types from offload (for example, list_models).

Error details stay in the database with three limits:

Content-hidden logs (content logging disabled with retain_content_in_object_storage on) keep no error details in the database. Their error details are offloaded with the rest of the payload, so these logs are left out of the error type, error code and status code breakdowns.
Raw provider bodies that error details carry when raw request/response storage is on are offloaded with the rest of the payload. Only the queryable error fields stay in the database row. Listing error_details in object_storage_exclude_fields keeps the whole value, bodies included, in the database.
Logs written by earlier versions with object storage on hold their error details only in object storage. They are not backfilled, so they stay out of the error breakdowns. Opening one of those logs still shows its error details.

Logs support S3, GCS, and Azure Blob Storage. Azure also supports skills files. Azure audit-log archival has not been verified. Local filesystem and data warehouse destinations are not supported.

Configuration via `config.json`

Amazon S3

Required IAM Permissions

The IAM user or role needs the following permissions on your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketAccess",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::bifrost-logs"
    },
    {
      "Sid": "ObjectAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:PutObjectTagging",
        "s3:GetObjectTagging"
      ],
      "Resource": "arn:aws:s3:::bifrost-logs/*"
    }
  ]
}

{
  "logs_store": {
    "enabled": true,
    "type": "sqlite",
    "config": {
      "path": "/app/data/logs.db"
    },
    "object_storage": {
      "type": "s3",
      "bucket": "env.AWS_S3_BUCKET",
      "region": "env.AWS_REGION",
      "access_key_id": "env.AWS_ACCESS_KEY_ID",
      "secret_access_key": "env.AWS_SECRET_ACCESS_KEY",
      "prefix": "bifrost/logs",
      "compress": true
    },
    "object_storage_exclude_fields": []
  }
}

S3 fields (all values support the env.<NAME> indirection to read from environment variables):

Field	Required	Notes
`type`	yes	Must be `"s3"`.
`bucket`	yes	Target bucket name.
`region`	yes	AWS region, e.g. `us-west-2`.
`access_key_id`	conditional	Required for static credentials. Omit when using `role_arn` or the default credential chain (IRSA, instance profile, env).
`secret_access_key`	conditional	Pairs with `access_key_id`.
`session_token`	optional	For temporary STS credentials.
`role_arn`	optional	Assume this role instead of using static keys.
`endpoint`	optional	Override for S3-compatible stores (MinIO, Cloudflare R2, Wasabi).
`force_path_style`	optional	Set `true` for most S3-compatible endpoints.
`prefix`	optional	Object key prefix. Defaults to `bifrost`.
`compress`	optional	Legacy gzip switch, same as `"compression": "gzip"`. Defaults to `false`.
`format`	optional	`json` (default) or `parquet`. See Payload format.
`compression`	optional	`none`, `gzip`, `snappy` or `zstd`. For `json` only `none` and `gzip` are allowed. Defaults to `none` for JSON (`gzip` when `compress` is `true`) and `snappy` for Parquet.

A live example is checked in at examples/configs/withobjectstorages3/config.json.

Google Cloud Storage

{
  "logs_store": {
    "enabled": true,
    "type": "sqlite",
    "config": {
      "path": "/app/data/logs.db"
    },
    "object_storage": {
      "type": "gcs",
      "bucket": "env.GCS_BUCKET",
      "credentials_json": "env.GCS_KEY",
      "project_id": "env.GCP_PROJECT_ID",
      "prefix": "bifrost/logs",
      "compress": true
    },
    "object_storage_exclude_fields": []
  }
}

GCS fields:

Field	Required	Notes
`type`	yes	Must be `"gcs"`.
`bucket`	yes	Target bucket name.
`credentials_json`	conditional	Service-account JSON (or `env.GCS_KEY` pointing at it). Omit when running on GKE with Workload Identity or any environment where Application Default Credentials work.
`credentials`	deprecated	Legacy alias for `credentials_json`. Use `credentials_json` in new configs.
`project_id`	optional	Useful when ADC cannot infer the project.
`prefix`	optional	Object key prefix. Defaults to `bifrost`.
`compress`	optional	Gzip payloads before upload. Defaults to `false`.

A live example is checked in at examples/configs/withobjectstoragegcs/config.json.

Azure Blob Storage

{
  "logs_store": {
    "enabled": true,
    "type": "sqlite",
    "config": { "path": "/app/data/logs.db" },
    "object_storage": {
      "type": "azure",
      "bucket": "bifrost-logs",
      "endpoint": "https://myaccount.blob.core.windows.net",
      "prefix": "bifrost",
      "compress": true
    }
  }
}

Field	Required	Notes
`type`	yes	Must be `"azure"`.
`bucket`	yes	Pre-existing container name, not the storage account name.
`endpoint`	identity mode	Blob service URL for `DefaultAzureCredential`.
`connection_string`	connection-string mode	Connection string or secret reference, such as `env.AZURE_STORAGE_CONNECTION_STRING`. Omit `endpoint`.
`prefix`	optional	Object key prefix. Defaults to `bifrost`.
`format`, `compression`, `compress`	optional	Same JSON/gzip and Parquet options as S3.

Set either endpoint or connection_string, not both. bucket and both authentication fields accept environment or vault references and must resolve to nonempty values. Identity mode supports managed identity, workload identity, or a service principal configured through AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET. Azure model-provider credentials are not reused. Grant Storage Blob Data Contributor on the container, or equivalent read/write/list/delete permissions. Attributes are stored as URL-encoded blob metadata, not index tags: no index-tag permissions are needed. URL-decode values when reading metadata outside Bifrost. See examples/configs/withobjectstorageazure/config.json for a connection-string example.

Payload format: JSON or Parquet

By default each offloaded payload is stored as a JSON object. Set format to parquet to store each payload as a single-row Parquet file instead, so the bucket can be queried directly with compatible data tools. Parquet is supported on S3 and Azure; GCS does not yet support it. The Athena examples below are S3-specific.

{
  "logs_store": {
    "object_storage": {
      "type": "s3",
      "bucket": "env.AWS_S3_BUCKET",
      "region": "env.AWS_REGION",
      "prefix": "bifrost/logs",
      "format": "parquet",
      "compression": "zstd"
    }
  }
}

Format and compression are separate settings:

`format`	`compression`	Object key	Stored body
`json`	`none` (default)	`<prefix>/logs/YYYY/MM/DD/HH/<id>.json.gz`	Plain JSON with no `Content-Encoding`. The key keeps the `.json.gz` suffix even though the body is not compressed.
`json`	`gzip`	`<prefix>/logs/YYYY/MM/DD/HH/<id>.json.gz`	Gzipped JSON, stored with `Content-Encoding: gzip`.
`parquet`	`none`, `gzip`, `snappy` (default), `zstd`	`<prefix>/logs/YYYY/MM/DD/HH/<id>.parquet`	Parquet file using that page codec. No outer gzip or `Content-Encoding` is added, so the file stays readable by Parquet tooling.

For json, the legacy compress: true is the same as compression: gzip. MCP tool logs, A2A agent logs and Relay Logs follow the same layout under mcp-logs/, agent-logs/ and relay-logs/. Each Parquet file holds one row. Every payload field becomes a nullable UTF8 column named after its database column (input_history, output_message, raw_request, token_usage, …). String values are stored as is; nested values (for example MCP tool arguments) are stored as JSON text, and their column names are listed in the file’s bifrost.json_columns key-value metadata. Objects are uploaded with Content-Type: application/vnd.apache.parquet. Switching formats, in either direction, does not rewrite existing objects. Bifrost keeps serving logs written before the switch: when the object in the configured format is missing, it reads the object in the other format at the same path (.json.gz or .parquet), and deleting a log removes both. A log updated after the switch (for example, a streamed response finishing) is rewritten in the configured format, and the copy in the old format is deleted, so each log keeps a single object. To query request-log payloads from Athena in a bucket that has only ever been written with format: parquet:

CREATE EXTERNAL TABLE bifrost_log_payloads (
  `timestamp`    string,
  provider       string,
  model          string,
  status         string,
  input_history  string,
  output_message string,
  params         string,
  token_usage    string,
  raw_request    string,
  raw_response   string
)
STORED AS PARQUET
LOCATION 's3://<bucket>/<prefix>/logs/';

Columns that a given log does not have are read as NULL. The log ID is the object’s file name, available in Athena as regexp_extract("$path", '([^/]+)\.parquet$', 1), for joining back to the logs database.

Athena reads every file under a table’s LOCATION and does not filter by file extension. In a bucket that switched from json to parquet, the hour folders written before the switch hold .json.gz objects, and a log from before the switch that is updated afterwards is rewritten as .parquet in its original hour folder. So pre-switch folders can hold both formats, and pointing the table above at logs/ fails on the JSON files.

For a bucket that switched formats, use partition projection over the hour folders and start the range at the first full hour after the switch, so Athena never lists a folder that can hold JSON:

CREATE EXTERNAL TABLE bifrost_log_payloads (
  `timestamp`    string,
  provider       string,
  model          string,
  status         string,
  input_history  string,
  output_message string,
  token_usage    string
)
PARTITIONED BY (hour string)
STORED AS PARQUET
LOCATION 's3://<bucket>/<prefix>/logs/'
TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.hour.type'        = 'date',
  'projection.hour.format'      = 'yyyy/MM/dd/HH',
  'projection.hour.range'       = '<first full hour after the switch, e.g. 2026/10/14/00>,NOW',
  'projection.hour.interval'    = '1',
  'projection.hour.interval.unit' = 'HOURS',
  'storage.location.template'   = 's3://<bucket>/<prefix>/logs/${hour}/'
);

Payloads from before the switch stay readable through Bifrost but are left out of this table. Engines that can filter by file name do not need the cutoff. For example, DuckDB can read read_parquet('s3://<bucket>/<prefix>/logs/**/*.parquet'), and Spark can use

spark.read.option("pathGlobFilter", "*.parquet").option("recursiveFileLookup", "true").parquet("s3://<bucket>/<prefix>/logs/")

Choosing what stays in the database vs. what is offloaded

By default, every offloadable payload field is uploaded to object storage. object_storage_exclude_fields lets you pin specific fields to the database only, so they are never written to the bucket. Listed fields are not dropped - they continue to live in the logs DB row as before. Everything not listed is offloaded. Values must be database column names (not JSON paths). Common choices:

Column	Contents
`raw_request`	The verbatim provider request body.
`raw_response`	The verbatim provider response body.
`input_history`	The full conversation sent to the model.
`output_message`	The model’s primary output message.

{
  "logs_store": {
    "object_storage": { "...": "..." },
    "object_storage_exclude_fields": [
      "raw_request",
      "raw_response"
    ]
  }
}

Unknown column names are silently ignored, so a typo will not error - it will just leave that field on the default (offloaded) path. Reference tests covering this behaviour live at framework/logstore/hybrid_test.go. Typical use cases:

Data residency: keep raw prompts and responses inside the DB (which may itself sit inside a controlled VPC) while still benefiting from offload for less sensitive fields.
Operational queries: keep input_history in the DB so SQL queries and the UI’s search can run against the full conversation without paying an object-fetch round trip.

Excluding request types

object_storage_exclude_request_types keeps every log of the listed request types out of the bucket. Use it for traffic you do not need in your data lake, such as model listing calls. Those logs are still written to the logs DB in full, so they keep showing up in the UI and API. Values are request type names as they appear in the log object field, for example list_models, embedding, chat_completion or chat_completion_stream. Streaming and non-streaming variants are separate types, so list both if you want to exclude both.

{
  "logs_store": {
    "object_storage": { "...": "..." },
    "object_storage_exclude_request_types": ["list_models"]
  }
}

Hidden logs are the exception. If content logging is disabled and retain_content_in_object_storage is on, logs of an excluded request type are still offloaded as hidden content, so that content stays out of the DB.

This is not the same as client.hidden_request_types. That setting hides request types from the logs view, dashboard, and log read APIs. It does not change what is written to the DB or uploaded to object storage.

Guardrail redaction

When Enterprise guardrail redaction is enabled, object storage receives the same persisted log payload fields that Bifrost logs use. That means redacted fields are offloaded as redacted fields. For reversible modes such as logs_only and runtime_reversible, the offloaded payload contains placeholders like [EMAIL-1]. The reveal mapping stays with the Bifrost log row and is not written into the offloaded object. If disable_content_logging is enabled, Bifrost skips content logging and does not persist reveal data. For the full mode matrix, see Guardrail Redaction.

Retention policy

Bifrost ships a background log cleaner that deletes old logs on a fixed schedule.

Configuration

Retention is configured on the client config, not on logs_store. The active key is client_config.log_retention_days:

{
  "client_config": {
    "log_retention_days": 30
  }
}

Aspect	Value
Default	`365` days
Minimum	`1` day (enforced by validator)
Disable	Set to `0` (cleanup is skipped entirely)
Cleanup cadence	24 hours after each pass finishes, plus a random jitter of 15-30 minutes
Startup behaviour	A cleanup pass runs immediately when Bifrost starts
Delete batch size	5,000 rows per query
Per-pass timeout	6 hours

Source: framework/logstore/cleaner.go and transports/bifrost-http/server/server.go.

What gets deleted

The cleaner deletes database rows older than now - retention_days (UTC). It deletes from the main Log table as well as the MCP tool logs table.

What does NOT get deleted automatically

The retention cleaner deletes database rows, not stored payloads. Configure lifecycle rules for logs/, mcp-logs/, agent-logs/, and relay-logs/ under your configured prefix, with retention longer than log_retention_days to cover the cleaner’s 24-hour interval, 15-30 minute jitter, and pass duration. For example, use 32 days for a 30-day log_retention_days setting. Allow more time for cleanup backlogs, failures, or downtime: missing payloads leave database rows readable without their offloaded content. Do not expire skills/ or the whole container. Amazon S3 lifecycle rule (example)

{
  "Rules": [
    {
      "ID": "bifrost-logs-expire-32d",
      "Status": "Enabled",
      "Filter": { "Prefix": "bifrost/logs/" },
      "Expiration": { "Days": 32 }
    }
  ]
}

Apply with aws s3api put-bucket-lifecycle-configuration --bucket <bucket> --lifecycle-configuration file://lifecycle.json. See the AWS docs on object lifecycle management. Google Cloud Storage lifecycle rule (example)

{
  "lifecycle": {
    "rule": [
      {
        "action": { "type": "Delete" },
        "condition": {
          "age": 32,
          "matchesPrefix": ["bifrost/logs/"]
        }
      }
    ]
  }
}

Apply with gcloud storage buckets update gs://<bucket> --lifecycle-file=lifecycle.json. See the GCS docs on Object Lifecycle Management.

Azure operations

Configure Azure Blob lifecycle management for block blobs. Each prefixMatch includes the container name. For container bifrost-logs and prefix: bifrost, use:

bifrost-logs/bifrost/logs/
bifrost-logs/bifrost/mcp-logs/
bifrost-logs/bifrost/agent-logs/
bifrost-logs/bifrost/relay-logs/

Do not expire skills/ or the whole container. Early expiry makes historical payloads unreadable; soft delete and version retention can delay permanent removal. Restart Bifrost after changing storage configuration or credentials. The SDK refreshes identity tokens automatically, but does not reload configuration. Changing backends does not move existing data. Before switching:

Keep the destination’s object_storage.prefix identical to the source. Historical payload lookups use the currently configured prefix, so changing it makes objects copied with their original keys unreadable. For example, retain bifrost/logs when migrating from the S3/GCS examples above rather than adopting the Azure example’s bifrost.
Copy objects with their exact keys, Content-Type, Content-Encoding, and metadata.
Pause writes or synchronize remaining changes before restarting against the destination.
Check historical payload reads before removing the source.

Manual deletes still cascade

Single and batch deletes triggered explicitly (via the UI’s Delete log action or the underlying DeleteLog / DeleteLogs APIs) do remove the associated objects from S3, GCS, or Azure Blob Storage at the same time. Only the time-based retention sweep is bucket-blind.

Configuration via Helm

Use the chart’s storage.logsStore.objectStorage values. For Azure identity mode:

storage:
  logsStore:
    objectStorage:
      enabled: true
      type: azure
      bucket: bifrost-logs
      endpoint: https://myaccount.blob.core.windows.net
      prefix: bifrost
      compress: true

For connection-string authentication, replace endpoint with existingSecret naming a Kubernetes Secret. Its default key is connection-string; override it with connectionStringKey. See Helm storage for an example.

BYO config.json

If you prefer to manage the full config.json yourself, mount it as a Secret and point the chart at it. Use the same logs_store.object_storage blocks shown in the config.json section verbatim.

Configuration via the UI

The web UI does not currently expose a form for logs_store.object_storage. To enable payload offload:

Edit config.json (or the Helm values) using the snippets above.
Restart Bifrost so the new log store wiring takes effect.
Confirm new requests are landing in the bucket by browsing the configured prefix/ path.

Once configured, the existing Logs screen in the UI transparently fetches payloads from object storage when you open a log entry. No UI changes are needed on the read path.

Verifying the setup

Send a request through Bifrost.
Confirm a row appears in the logs DB (visible via the UI’s Logs screen).
List the bucket under your prefix/. You should see one or more objects per request, ending in .json.gz or, with format: parquet, .parquet.
Open the log entry in the UI. The payload pane should render the content fetched from object storage.

If the bucket stays empty, check the Bifrost logs for objectstore errors. The most common causes are missing credentials, a region/endpoint mismatch, or a bucket policy that blocks the credentials’ principal.

Getting Started

Moving from OSS

Release Cadence

Migration Guides

Features

Security

How it works

Configuration via `config.json`

Amazon S3

Required IAM Permissions

Google Cloud Storage

Azure Blob Storage

Payload format: JSON or Parquet

Choosing what stays in the database vs. what is offloaded

Excluding request types

Guardrail redaction

Retention policy

Configuration

What gets deleted

What does NOT get deleted automatically

Azure operations

Manual deletes still cascade

Configuration via Helm

BYO config.json

Configuration via the UI

Verifying the setup

​How it works

​Configuration via config.json

​Amazon S3

​Required IAM Permissions

​Google Cloud Storage

​Azure Blob Storage

​Payload format: JSON or Parquet

​Choosing what stays in the database vs. what is offloaded

​Excluding request types

​Guardrail redaction

​Retention policy

​Configuration

​What gets deleted

​What does NOT get deleted automatically

​Azure operations

​Manual deletes still cascade

​Configuration via Helm

​BYO config.json

​Configuration via the UI

​Verifying the setup

How it works

Configuration via `config.json`

Amazon S3

Required IAM Permissions

Google Cloud Storage

Azure Blob Storage

Payload format: JSON or Parquet

Choosing what stays in the database vs. what is offloaded

Excluding request types

Guardrail redaction

Retention policy

Configuration

What gets deleted

What does NOT get deleted automatically

Azure operations

Manual deletes still cascade

Configuration via Helm

BYO config.json

Configuration via the UI

Verifying the setup