OpenTelemetry Tracing

MinIO AIStor supports OpenTelemetry-based distributed tracing to capture and analyze request flows across the server. Traces can be streamed to an OTLP-compatible collector, recorded to a local file, or captured on-demand for troubleshooting.

Architecture

MinIO AIStor exports trace data using the OpenTelemetry Protocol (OTLP) over HTTP:

MinIO AIStor Server  →  OTLP Collector  →  Tracing Backend (Jaeger, Grafana Tempo, etc.)

Server-side configuration

Configure MinIO AIStor to continuously export traces to a collector using environment variables or mc admin config set.

See Telemetry Settings for the full environment variable reference.

Enable telemetry

export MINIO_TELEMETRY_TARGET_ENABLE="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT="http://otlp-collector.example.net:4318"
export MINIO_TELEMETRY_TARGET_TYPE="s3"
export MINIO_TELEMETRY_TARGET_SAMPLE_RATE="0.1"

Or using mc admin config set:

mc admin config set ALIAS telemetry_target \
  enable=on \
  endpoint=http://otlp-collector.example.net:4318 \
  type=s3 \
  sample_rate=0.1

Named targets

Configure multiple telemetry targets by appending a target name:

export MINIO_TELEMETRY_TARGET_ENABLE_jaeger="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT_jaeger="http://jaeger:4318"
export MINIO_TELEMETRY_TARGET_TYPE_jaeger="s3,internal"

export MINIO_TELEMETRY_TARGET_ENABLE_debug="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT_debug="http://debug-collector:4318"
export MINIO_TELEMETRY_TARGET_TYPE_debug="healing,scanner"

Trace types

MinIO AIStor supports the following trace types. Specify one or more as a comma-separated list in the MINIO_TELEMETRY_TARGET_TYPE setting.

Type	Description
`Admin`	Admin API calls
`BatchExpire`	Batch expiration operations
`BatchKeyRotation`	Batch key rotation operations
`BatchReplication`	Batch replication operations
`Bootstrap`	Cluster bootstrap events
`Decommission`	Decommission operations
`Formatting`	Drive formatting events
`FTP`	FTP/SFTP server calls
`Healing`	Healing operations
`IAM`	Identity and access management calls
`ILM`	Lifecycle management calls
`Internal`	Internal `.minio.sys/` operations
`KMS`	KMS interaction calls
`Object`	Object layer calls
`OS`	Go OS package calls
`Replication`	Replication worker events
`ReplicationResync`	Resync operations
`Rebalance`	Rebalance operations
`S3`	S3 API calls
`Scanner`	Scanner operations
`Storage`	Storage layer calls
`Tables`	AIStor Tables (Iceberg REST Catalog) API calls

Capitalization is ignored when specifying trace types.

On-demand trace capture

Use mc support telemetry to capture traces without configuring server-side settings.

Record traces to a file

mc support telemetry record ALIAS --type s3,internal --duration 5m

The trace data is saved as a compressed file that can be shared with MinIO support or analyzed locally.

Proxy traces in real-time

Stream traces to a local collector for live analysis:

mc support telemetry proxy ALIAS --type s3 --endpoint http://localhost:4318

Analyze traces

Use the built-in TUI analyzer to inspect recorded trace files:

mc support telemetry analyze trace-file.gz

Replay traces

Replay a previously recorded trace file to a collector:

mc support telemetry replay trace-file.gz --endpoint http://jaeger:4318

Jaeger integration example

Start Jaeger with OTLP support:

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/jaeger:latest

Configure MinIO AIStor:

export MINIO_TELEMETRY_TARGET_ENABLE="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT="http://jaeger:4318"
export MINIO_TELEMETRY_TARGET_TYPE="s3"
export MINIO_TELEMETRY_TARGET_SAMPLE_RATE="0.1"

Access the Jaeger UI at http://localhost:16686 to view traces.

Performance considerations

Sample rate	Impact	Use case
`0.01`	Minimal	Production monitoring
`0.1`	Low	General observability
`0.5`	Moderate	Debugging specific issues
`1.0`	High	Development or short-term debugging

Higher sample rates capture more complete trace data but increase CPU and network overhead. For production deployments, start with a sample rate of 0.01 to 0.1.

Privacy considerations

Trace data may contain:

S3 API paths including bucket and object key names
IAM user identifiers
Client IP addresses
Internal cluster communication details

Ensure your tracing backend enforces appropriate access controls and data retention policies.