OpenTelemetry Tracing

MinIO AIStor supports OpenTelemetry-based distributed tracing to capture and analyze request flows across the server. Traces can be streamed to an OTLP-compatible collector, recorded to a local file, or captured on-demand for troubleshooting.

MinIO AIStor exports trace data using the OpenTelemetry Protocol (OTLP) over HTTP:

MinIO AIStor Server  →  OTLP Collector  →  Tracing Backend (Jaeger, Grafana Tempo, etc.)

Configure MinIO AIStor to continuously export traces to a collector using environment variables or mc admin config set.

See Telemetry Settings for the full environment variable reference.

export MINIO_TELEMETRY_TARGET_ENABLE="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT="http://otlp-collector.example.net:4318"
export MINIO_TELEMETRY_TARGET_TYPE="s3"
export MINIO_TELEMETRY_TARGET_SAMPLE_RATE="0.1"

Or using mc admin config set:

mc admin config set ALIAS telemetry_target \
  enable=on \
  endpoint=http://otlp-collector.example.net:4318 \
  type=s3 \
  sample_rate=0.1

Configure multiple telemetry targets by appending a target name:

export MINIO_TELEMETRY_TARGET_ENABLE_jaeger="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT_jaeger="http://jaeger:4318"
export MINIO_TELEMETRY_TARGET_TYPE_jaeger="s3,internal"

export MINIO_TELEMETRY_TARGET_ENABLE_debug="on"
export MINIO_TELEMETRY_TARGET_ENDPOINT_debug="http://debug-collector:4318"
export MINIO_TELEMETRY_TARGET_TYPE_debug="healing,scanner"

MinIO AIStor supports the following trace types. Specify one or more as a comma-separated list in the MINIO_TELEMETRY_TARGET_TYPE setting.

Capitalization is ignored when specifying trace types.

Use mc support telemetry to capture traces without configuring server-side settings.

mc support telemetry record captures traces to a file. Choose the form that matches what you plan to do with the capture.

To send the capture to MinIO, record it with the default encryption:

mc support telemetry record --type s3 --type internal --duration 300 ALIAS

This writes metrics-<TIMESTAMP>.enc to the working directory, encrypted with the MinIO public certificate. Only MinIO can read that file. Use mc support upload to attach it to a SUBNET issue.

To analyze the capture yourself, record it unencrypted:

mc support telemetry record --type s3 --type internal --duration 300 --unencrypted --out mytrace.bin ALIAS

Both forms take the same filters:

• --duration is a whole number of seconds. Omit it to record until you press Ctrl+C.
• Repeat --type to capture more than one trace type. A comma-separated list is read as a single unrecognized type, and the command silently falls back to s3.

Stream traces to a local collector for live analysis. The collector URL is the second positional argument, after the alias, rather than a flag:

mc support telemetry proxy --type s3 ALIAS http://localhost:4318

Open a recorded trace file in the built-in interactive terminal analyzer:

mc support telemetry analyze mytrace.bin

To analyze a file you encrypted with your own key pair, pass the matching private key with --private-key.

Replay a previously recorded trace file to a collector. As with proxy, the collector URL is the second positional argument, here after the input file:

mc support telemetry replay mytrace.bin http://jaeger:4318

1. Start Jaeger with OTLP support:

   docker run -d --name jaeger \
     -p 16686:16686 \
     -p 4318:4318 \
     jaegertracing/jaeger:latest
   

2. Configure MinIO AIStor:

   export MINIO_TELEMETRY_TARGET_ENABLE="on"
   export MINIO_TELEMETRY_TARGET_ENDPOINT="http://jaeger:4318"
   export MINIO_TELEMETRY_TARGET_TYPE="s3"
   export MINIO_TELEMETRY_TARGET_SAMPLE_RATE="0.1"
   

3. Access the Jaeger UI at http://localhost:16686 to view traces.

Higher sample rates capture more complete trace data but increase CPU and network overhead. For production deployments, start with a sample rate of 0.01 to 0.1.

Trace data may contain:

• S3 API paths including bucket and object key names
• IAM user identifiers
• Client IP addresses
• Internal cluster communication details

Ensure your tracing backend enforces appropriate access controls and data retention policies.