OpenTelemetry Integration

This page describes how to configure OpenTelemetry integration for the MinIO AIStor Operator. When enabled, the Operator exports structured log data to an OpenTelemetry Protocol (OTLP) collector for centralized logging, analysis, and correlation with other telemetry signals.

The MinIO AIStor Operator integrates OpenTelemetry through a klog bridge that forwards all Operator log messages to an OTLP collector. This integration provides structured, machine-readable logs enriched with Kubernetes metadata for correlation with other observability signals.

When you configure the OTEL_EXPORTER_OTLP_ENDPOINT environment variable on the Operator Deployment, the Operator:

1. Initializes an OTLP log exporter using the configured protocol (gRPC or HTTP).
2. Creates a log provider with Kubernetes resource attributes.
3. Bridges klog output to the OpenTelemetry log provider.
4. Exports batched log records to the configured collector endpoint.

If the collector is unavailable at startup, the Operator logs a warning and continues operation using standard klog output. This ensures the Operator remains functional even if the observability infrastructure is temporarily unavailable.

Each log record exported to the OTLP collector includes:

Severity levels mapped from klog verbosity:

Resource attributes identifying the Operator instance:

The Kubernetes attributes require configuring the corresponding pod metadata environment variables using the Kubernetes downward API. See Pod metadata for OTLP for configuration details.

To enable OpenTelemetry logging, add the required environment variables to the Operator Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: object-store-operator
  namespace: aistor
spec:
  template:
    spec:
      containers:
      - name: operator
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector.observability.svc:4317"

operator:
  env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability.svc:4317"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "grpc"
  - name: POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: POD_NODENAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

apiVersion: apps/v1
kind: Deployment
metadata:
  name: object-store-operator
  namespace: aistor
spec:
  template:
    spec:
      containers:
      - name: operator
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector.observability.svc:4317"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "grpc"
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NODENAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

The Operator supports three OTLP protocols for log export:

Set the protocol using the OTEL_EXPORTER_OTLP_PROTOCOL environment variable. If not specified, the Operator defaults to grpc.

The OTLP collector must support the OTLP logs protocol. Common collectors include:

• OpenTelemetry Collector with the OTLP receiver.
• Grafana Alloy (formerly Grafana Agent).
• Cloud provider collectors (AWS CloudWatch, Google Cloud Logging, Azure Monitor).
• Observability platforms with native OTLP support (Datadog, Honeycomb, Splunk).

Configure your collector to receive OTLP logs on the appropriate port and protocol. The following example shows an OpenTelemetry Collector configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  # Configure your backend exporter here
  logging:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

After configuring OpenTelemetry, verify the integration is working:

1. Check the Operator logs for initialization messages.

   kubectl logs -n aistor deployment/object-store-operator | grep -i opentelemetry
   

   Successful initialization output resembles the following:

   OpenTelemetry enabled, initializing OTLP log exporter (endpoint: http://otel-collector.observability.svc:4317, protocol: grpc)
   OpenTelemetry log provider initialized
   OpenTelemetry klog bridge registered successfully
   

2. Query your observability backend for logs with service.name="aistor-operator".

3. Verify Kubernetes attributes are present by filtering on k8s.namespace.name or k8s.pod.name.

The OTEL_EXPORTER_OTLP_ENDPOINT environment variable is not set or is empty. Verify the environment variable is correctly configured in the Operator Deployment.

The Operator could not connect to the OTLP collector. Verify:

• The collector endpoint URL is correct and reachable from the Operator pod.
• The collector is running and accepting connections on the configured port.
• Network policies allow traffic from the Operator namespace to the collector.

The Operator continues operating with standard klog output when OTLP initialization fails.

The pod metadata environment variables are not configured. Add the POD_NAMESPACE, POD_NAME, and POD_NODENAME environment variables using the Kubernetes downward API as shown in the Complete configuration tab above.

Reduce log verbosity or increase the batch processor timeout in your collector configuration. The Operator uses batched log export to reduce network overhead, but high-volume debug logging can increase memory usage.

• Operator Environment Variables - Complete reference for all Operator environment variables including OpenTelemetry settings
• Telemetry Settings - Configure OpenTelemetry for individual Object Store deployments
• OpenTelemetry documentation - Official OpenTelemetry project documentation