OpenTelemetry Integration

This page describes how to configure OpenTelemetry integration for the MinIO AIStor Operator. When enabled, the Operator exports structured log data to an OpenTelemetry Protocol (OTLP) collector for centralized logging, analysis, and correlation with other telemetry signals.

How OpenTelemetry logging works

The MinIO AIStor Operator integrates OpenTelemetry through a klog bridge that forwards all Operator log messages to an OTLP collector. This integration provides structured, machine-readable logs enriched with Kubernetes metadata for correlation with other observability signals.

When you configure the OTEL_EXPORTER_OTLP_ENDPOINT environment variable on the Operator Deployment, the Operator:

Initializes an OTLP log exporter using the configured protocol (gRPC or HTTP).
Creates a log provider with Kubernetes resource attributes.
Bridges klog output to the OpenTelemetry log provider.
Exports batched log records to the configured collector endpoint.

If the collector is unavailable at startup, the Operator logs a warning and continues operation using standard klog output. This ensures the Operator remains functional even if the observability infrastructure is temporarily unavailable.

Log record structure

Each log record exported to the OTLP collector includes:

Severity levels mapped from klog verbosity:

klog level	OTLP severity
Info (V=0)	Info
V=1	Debug
V=2	Debug2
V=3	Debug3
V=4+	Debug4
Error	Error

Resource attributes identifying the Operator instance:

Attribute	Description
`service.name`	Always `aistor-operator`
`service.version`	Operator version and commit ID
`host.name`	Pod hostname
`k8s.namespace.name`	Namespace where the Operator runs
`k8s.pod.name`	Operator pod name
`k8s.node.name`	Node where the Operator pod runs

The Kubernetes attributes require configuring the corresponding pod metadata environment variables using the Kubernetes downward API. See Configure OpenTelemetry logging for a worked example that sets all three.

Configure OpenTelemetry logging

To enable OpenTelemetry logging, add the required environment variables to the Operator Deployment.

The following example configures the Operator to export logs to an OTLP collector using gRPC:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: object-store-operator
  namespace: aistor
spec:
  template:
    spec:
      containers:
      - name: operator
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector.observability.svc:4317"

When deploying with Helm, add the environment variables to your values file:

global:
  operator:
    extraEnv:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://otel-collector.observability.svc:4317"
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: "grpc"
    - name: POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: POD_NODENAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

For full observability with Kubernetes resource attributes, configure the downward API environment variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: object-store-operator
  namespace: aistor
spec:
  template:
    spec:
      containers:
      - name: operator
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector.observability.svc:4317"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "grpc"
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NODENAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

Protocol options

The Operator supports three OTLP protocols for log export:

Protocol	Port (typical)	Use case
`grpc`	4317	Default. Lower overhead for high-volume logging.
`http/protobuf`	4318	HTTP transport with Protocol Buffer encoding. Use when gRPC is blocked by firewalls or proxies.
`http/json`	4318	HTTP transport with JSON encoding. Useful for debugging or for human-readable payloads.

Set the protocol using the OTEL_EXPORTER_OTLP_PROTOCOL environment variable. If not specified, the Operator defaults to grpc.

Collector requirements

The OTLP collector must support the OTLP logs protocol. Common collectors include:

OpenTelemetry Collector with the OTLP receiver.
Grafana Alloy (formerly Grafana Agent).
Cloud provider collectors (AWS CloudWatch, Google Cloud Logging, Azure Monitor).
Observability platforms with native OTLP support (Datadog, Honeycomb, Splunk).

Configure your collector to receive OTLP logs on the appropriate port and protocol. The following example shows an OpenTelemetry Collector configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  # Configure your backend exporter here
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]

Verify OpenTelemetry logging

After configuring OpenTelemetry, verify the integration is working:

Check the Operator logs for initialization messages.

kubectl logs -n aistor deployment/object-store-operator | grep -i opentelemetry

Successful initialization output resembles the following:

OpenTelemetry enabled, initializing OTLP log exporter (endpoint: http://otel-collector.observability.svc:4317, protocol: grpc)
OpenTelemetry log provider initialized
OpenTelemetry klog bridge registered successfully

Query your observability backend for logs with service.name="aistor-operator".
Verify Kubernetes attributes are present by filtering on k8s.namespace.name or k8s.pod.name.

Troubleshooting

Operator logs show “OpenTelemetry not configured”

The OTEL_EXPORTER_OTLP_ENDPOINT environment variable is not set or is empty. Verify the environment variable is correctly configured in the Operator Deployment.

Operator logs show “Failed to initialize OpenTelemetry log provider”

The Operator could not connect to the OTLP collector. Verify:

The collector endpoint URL is correct and reachable from the Operator pod.
The collector is running and accepting connections on the configured port.
Network policies allow traffic from the Operator namespace to the collector.

The Operator continues operating with standard klog output when OTLP initialization fails.

Missing Kubernetes attributes in log records

The pod metadata environment variables are not configured. Add the POD_NAMESPACE, POD_NAME, and POD_NODENAME environment variables using the Kubernetes downward API as shown in the Complete configuration tab above.

High memory usage with verbose logging

Reduce log verbosity or increase the batch processor timeout in your collector configuration. The Operator uses batched log export to reduce network overhead, but high-volume debug logging can increase memory usage.

Operator Environment Variables - Reference for the other Operator environment variables
Telemetry Settings - Configure OpenTelemetry for individual Object Store deployments
OpenTelemetry documentation - Official OpenTelemetry project documentation

OpenTelemetry Integration

On this page