Enable RDMA

Configure the MinIO AIStor Operator and Object Store Helm charts to move object data over RDMA (Remote Direct Memory Access) instead of TCP.

This page covers only the Kubernetes layer: the Helm values that enable RDMA, the Pod changes the Operator makes, and how to check the result with kubectl. The fabric itself is not a Kubernetes concern. Configure the switch, the host NIC, and the kernel first, on every node that runs an Object Store Pod, by following Configure the RDMA fabric.

Complete these steps on the hosts first. They apply to any RDMA deployment and are documented in the operations runbook:

1. Configure the RDMA fabric — hardware, switch, host NIC, and PCIe settings, including Priority Flow Control.
2. Confirm each node reports an RDMA device with an ACTIVE port, as described in that page.

The remaining prerequisites are specific to Kubernetes.

• A Container Network Interface (CNI) plugin that permits Pods on the host network. The Object Store Pods run in the host network namespace.

• A namespace that permits privileged Pods. With Pod Security Admission enforcing, which is the default since Kubernetes 1.25, label the namespace:

  kubectl label namespace OBJECT-STORE-NAMESPACE \
    pod-security.kubernetes.io/enforce=privileged \
    pod-security.kubernetes.io/audit=privileged \
    pod-security.kubernetes.io/warn=privileged --overwrite
  

  Any other admission controller, such as Gatekeeper or Kyverno, must also permit privileged, hostNetwork, hostPath, and the IPC_LOCK capability.

• At least as many RDMA-capable nodes as the pool has servers. The Operator injects a required Pod anti-affinity that allows one server per node, so a pool larger than the node count never schedules.

RDMA is compiled out of the standard AIStor Server build. Pin the RDMA build in the Object Store chart values, as shown in Configure the Object Store chart.

OpenShift rejects the combination of privileged and hostNetwork under its default security context constraints. Bind the privileged SCC to the Object Store service account, which the Operator names OBJECT-STORE-NAME-sa:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: aistor-rdma-privileged
  namespace: OBJECT-STORE-NAMESPACE
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
  - kind: ServiceAccount
    name: OBJECT-STORE-NAME-sa
    namespace: OBJECT-STORE-NAMESPACE

The Operator creates the ServiceAccount when it reconciles the Object Store. Apply the RoleBinding first if you prefer, because it binds by name.

One decision shapes the rest of the configuration: does the RDMA adapter carry the IP address that the Object Store binds?

RoCE v2 needs a GID that matches the source address, and a GID exists only for an address configured on the netdev that backs the RDMA device. An address on a bridge, a bond, or an overlay above the adapter creates no IPv4 GID, and RDMA cannot bind it. Configure the RDMA fabric shows how to read the GID table.

Bridged node addresses are the norm on OpenShift and are common on kubeadm clusters that use a bond.

When objectStore.rdma.enabled is true, the Operator changes each pool StatefulSet:

Declare affinity.podAntiAffinity on a pool to replace the injected rule.

Inter-node RDMA pins buffer pools in physical memory on every node. The default is 256 buffers of 64 MiB, which reserves 16 GiB per node. Size the nodes for it, or reduce it through the Object Store environment:

objectStore:
  env:
  - name: MINIO_RDMA_POOL_COUNT
    value: "64"
  - name: MINIO_RDMA_POOL_BUF_SIZE
    value: "67108864"

Set objectStore.rdma.internode to false to keep the client-to-server path while disabling inter-node shard transfer. For the full list of settings, see RDMA settings.

The Operator itself needs no RDMA-specific values. Install or upgrade the aistor-operator chart at 6.0.0 or later:

helm upgrade --install aistor minio/aistor-operator \
  -n aistor --create-namespace \
  -f aistor-operator-values.yaml

Confirm the chart version:

helm list -n aistor

NAME    NAMESPACE   REVISION   CHART                   APP VERSION
aistor  aistor      1          aistor-operator-6.0.0   v20260721224148.0.0

RDMA is enabled per Object Store through objectStore.rdma in the aistor-objectstore chart.

Use this topology when the RDMA adapter already carries the node IP. Nothing needs to change on the host beyond the fabric configuration:

objectStore:
  image:
    repository: quay.io/minio/aistor/minio
    tag: RELEASE.2026-08-07T18-34-35Z.rdma
  rdma:
    enabled: true
    # internode: true
    # peerPort: 5555
    # devicePath: /dev/infiniband
  pools:
  - name: pool-0
    servers: 4
    volumesPerServer: 10
    # Restrict scheduling to RDMA-capable nodes. node-feature-discovery, which
    # ships with the NVIDIA GPU Operator, applies this label.
    nodeSelector:
      feature.node.kubernetes.io/rdma.capable: "true"
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: STORAGE-CLASS-NAME
        resources:
          requests:
            storage: 2Ti

Apply the values:

helm upgrade --install -n OBJECT-STORE-NAMESPACE OBJECT-STORE-NAME minio/aistor-objectstore \
  -f aistor-objectstore-values.yaml

Use this topology when the fabric adapter has its own address, separate from the node IP. The Operator reads each server’s fabric address from a Node label and pins it into the generated MinIO configuration.

Label every node that runs a server:

kubectl label node NODE-NAME aistor.min.io/rdma-ip=198.51.100.11 --overwrite

Add a second label when the node has a second fabric adapter:

kubectl label node NODE-NAME aistor.min.io/rdma-ip2=203.0.113.11 --overwrite

Then name the labels in the chart values:

objectStore:
  image:
    repository: quay.io/minio/aistor/minio
    tag: RELEASE.2026-08-07T18-34-35Z.rdma
  rdma:
    enabled: true
    fabric:
      nodeAddressLabel: aistor.min.io/rdma-ip
      # Optional. Each node then advertises two addresses and the server
      # enables multi-NIC inter-node routing.
      secondaryNodeAddressLabel: aistor.min.io/rdma-ip2
  pools:
  - name: pool-0
    servers: 4
    volumesPerServer: 10
    volumeClaimTemplate:
      metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: STORAGE-CLASS-NAME
        resources:
          requests:
            storage: 2Ti

The Operator pins the fabric addresses only after every server Pod is scheduled and every node carries the label. Until then it leaves the cluster DNS addresses in place, and the Object Store stays healthy over TCP. Resolution is sticky: a transient failure reuses the addresses already in the generated Secret rather than reverting and restarting the server.

Multi-NIC mode requires an AIStor Server image that supports MinIO configuration version 3. An older image produces an UnsupportedVersion event on the Object Store instead of silently ignoring the fabric addresses.

Multus cannot bring up a host adapter and cannot supply the RDMA interface for an Object Store. It attaches interfaces to a Pod network namespace, and RDMA Pods run on the host network, so a NetworkAttachmentDefinition on them has no effect. The Operator reads fabric addresses from Node labels and has no code path that consumes a NetworkAttachmentDefinition.

Multus is useful for workloads that are not managed by the Operator and need to reach the fabric subnet without host networking. A macvlan attachment gives a Pod an address on the fabric but not RDMA verbs, which additionally require rdma-cni with an SR-IOV virtual function.

Work through Validate the RDMA deployment for the adapter, fabric, server, and metric checks. Run the host-level commands in that page inside an Object Store Pod, because the Pods use the host network and see the adapter directly:

kubectl -n OBJECT-STORE-NAMESPACE exec OBJECT-STORE-NAME-pool-0-0 -c minio -- \
  cat /sys/class/infiniband/RDMA-DEVICE/ports/PORT-NUMBER/counters/port_xmit_packets

Replace RDMA-DEVICE and PORT-NUMBER with the adapter and the active port reported by ibv_devinfo in Confirm the host sees an RDMA device. The device name depends on the driver, and a dual-port adapter can have its fabric link on either port.

Three checks have no equivalent outside Kubernetes.

Confirm the Object Store is healthy and that one Pod runs per node:

kubectl -n OBJECT-STORE-NAMESPACE get objectstore OBJECT-STORE-NAME
kubectl -n OBJECT-STORE-NAMESPACE get pods -o wide

Confirm the addresses the Operator pinned, which is the check that proves multi-NIC mode took effect:

kubectl -n OBJECT-STORE-NAMESPACE get secret OBJECT-STORE-NAME-generated \
  -o jsonpath='{.data.config\.yaml}' | base64 -d

pools:
    - nodes:
        - addresses:
            - https://198.51.100.11:9000
            - https://203.0.113.11:9000
          path: /export{0...3}

Each server has its own entry carrying that node’s own addresses. An entry that repeats one address, or the cluster DNS form, means the Operator has not pinned the fabric yet.

Confirm the annotation that rolls the Pods when a fabric address changes:

kubectl -n OBJECT-STORE-NAMESPACE get statefulset OBJECT-STORE-NAME-pool-0 \
  -o jsonpath='{.spec.template.metadata.annotations.aistor\.min\.io/rdma-config-hash}'

The Operator scrapes the two RDMA metric groups automatically when RDMA is enabled. Read them with the AIStor Client:

mc admin prometheus metrics myaistor api --api-version v3 | grep rdma
mc admin prometheus metrics myaistor system --api-version v3 | grep rdma

The two RDMA paths are benchmarked differently, and conflating them is the usual reason a run appears to show no RDMA activity.

Inter-node RDMA is driven by any S3 workload. The client speaks ordinary S3 over TCP, and the fabric carries the erasure-coded shards between servers. The client needs no RDMA adapter, no host networking, no device mount, and no custom image, which is what makes this the path to benchmark first.

Three ways to run the benchmark are described below, in increasing order of setup cost. They produce the same result, so choose by what the cluster already has.

The fewest moving parts, and the one to reach for first. Install WARP on any host that can reach the Object Store, following WARP bare-metal installation. The standard package is enough, because this path needs no RDMA on the client.

Point the run at the Object Store Service or at the node addresses:

warp put --host=OBJECT-STORE-HOST:9000 --tls \
  --access-key=ACCESS_KEY --secret-key=SECRET_KEY \
  --obj.size=256MiB --objects=200 --concurrent=32 --duration=5m \
  --bucket=warp-internode

Add --warp-client to drive several hosts at once, as the WARP documentation describes. The client network path is irrelevant to what the fabric carries, because inter-node RDMA is a server-side concern.

Use this when the benchmark should run inside the cluster and you do not want to enable another operator. The warp chart runs a set of client Pods and a job that drives them:

helm repo add minio https://helm.min.io
helm install warp-internode minio/warp -n OBJECT-STORE-NAMESPACE -f warp-values.yaml

replicaCount: 4
configFile: |
  warp:
    api: v1
    benchmark: put
    remote:
      access-key: ACCESS_KEY
      secret-key: SECRET_KEY
      host:
        - 'OBJECT-STORE-NAME-pool-0-0.OBJECT-STORE-NAME-hl.OBJECT-STORE-NAMESPACE.svc.cluster.local:9000'
        - 'OBJECT-STORE-NAME-pool-0-1.OBJECT-STORE-NAME-hl.OBJECT-STORE-NAMESPACE.svc.cluster.local:9000'
      tls: true
    params:
      duration: 5m
      concurrent: 32
      objects: 200
      obj:
        size: 256MiB

Read the result from the job logs:

kubectl -n OBJECT-STORE-NAMESPACE logs job/warp-internode

Use this when you want the benchmark declared as a Kubernetes resource alongside the Object Store. The Warp operator is disabled by default, so enable it in the aistor-operator chart values and upgrade the release:

operators:
  warp:
    disabled: false

The Warp resource takes the Object Store by name, so it needs no endpoint or credentials:

apiVersion: aistor.min.io/v1alpha1
kind: Warp
metadata:
  name: internode-rdma
  namespace: OBJECT-STORE-NAMESPACE
spec:
  type: put
  clients: 4
  duration: 5m
  objectStore:
    namespace: OBJECT-STORE-NAMESPACE
    name: OBJECT-STORE-NAME
  args:
    # Large enough that object size divided by data drives clears
    # MINIO_RDMA_MIN_SIZE.
    obj.size: "256MiB"
    objects: "200"
    concurrent: "32"

kubectl apply -f warp-internode.yaml
kubectl -n OBJECT-STORE-NAMESPACE get warp internode-rdma -o jsonpath='{.status.results}'

A run that fell back to TCP looks exactly like a successful one in the WARP output, so read the metrics instead. Inter-node traffic is reported under the system metric type, on the /system/network/internode/rdma path.

Point the AIStor Client at the Object Store, then read the counters:

mc alias set myaistor https://OBJECT-STORE-HOST:9000 ACCESS_KEY SECRET_KEY
mc admin prometheus metrics myaistor system --api-version v3 | grep internode_rdma

These are the counters that describe inter-node traffic:

A put benchmark moves the write counters. Read the values immediately before and after the run and compare the deltas, because every counter is cumulative since the server started. Each series also carries a server label, so sum across the servers for a cluster total.

Two more counters explain a result that looks wrong:

Write counters that stay flat while the benchmark runs mean the shards never reached the fabric. Check the object size first, then work through Troubleshoot RDMA.

For the full list, including the remaining per-NIC counters, see Metrics version 3.

An absolute throughput figure means little on its own, because the drives or the client may be the bottleneck before the fabric is. Run the same benchmark twice and change only the transport.

Set objectStore.rdma.internode to false for the baseline, wait for the rollout, run the benchmark, then set it back to true and run it again. Each change rolls the Pods, so wait for the rollout to finish between runs:

kubectl -n OBJECT-STORE-NAMESPACE rollout status statefulset/OBJECT-STORE-NAME-pool-0

Read the metrics immediately before and after each run and compare the deltas. That is what proves which transport carried the payload.

The client-to-server path needs an RDMA-aware client. WARP 1.7.0 and later links libs3rdma, the same library the server uses for this path.

Run this client on a bare-metal host with its own RDMA adapter on the fabric, rather than as a Pod. Install it with the packages described in WARP bare-metal installation, which provides both the binary and libs3rdma.

The client host needs the same fabric configuration as a server node: an adapter with an ACTIVE port and an IPv4 GID, /dev/infiniband/uverbs0 and /dev/infiniband/rdma_cm readable by the user running WARP, a driver file in /etc/libibverbs.d/, and an unlimited locked-memory limit.

Point the run at the fabric addresses rather than the cluster Service DNS name. Traffic sent to the Service leaves over the primary adapter and never exercises the RDMA path. Pin the device to the one that carries the fabric address:

export S3RDMA_DEVICE=mlx5_0
warp get --host=198.51.100.11:9000,198.51.100.12:9000 --host-select=roundrobin --tls \
  --access-key=ACCESS_KEY --secret-key=SECRET_KEY --rdma=cpu \
  --obj.size=64MiB --objects=500 --concurrent=32 --duration=5m \
  --bucket=warp-rdma

Omit --tls when the endpoint serves plain HTTP. Do not add --insecure when a transfer fails, because it does not apply to the RDMA path and hides a control-plane certificate error while leaving the RDMA error unchanged.

--rdma=cpu transfers into host memory and needs no GPU. --rdma=gpu transfers into GPU device memory over GPUDirect RDMA, which needs an NVIDIA GPU with a compatible driver and CUDA runtime. See GPU for the GPU, PCIe, and driver requirements.

To drive several client hosts, start warp client on each one with S3RDMA_DEVICE set in its own environment, because the driver does not propagate the variable. Every daemon must run the same WARP build as the driver.

A run that fell back to TCP looks the same in the WARP output, so read the metrics instead. Client-to-server traffic is reported under the api metric type, on the /api/rdma path, and is counted separately from the inter-node path:

mc alias set myaistor https://198.51.100.11:9000 ACCESS_KEY SECRET_KEY
mc admin prometheus metrics myaistor api --api-version v3 | grep api_rdma

These are the counters that describe client-to-server traffic:

Read the values immediately before and after the run and compare the deltas, because every counter is cumulative since the server started. Each series carries a server label, so sum across the servers for a cluster total.

Counters that stay flat mean the client never used the RDMA path, even when the transfers succeeded. The usual causes are an endpoint that is not a fabric address, a client without the issuing certificate authority in its system trust store, and a WARP build older than 1.7.0.

A client can also confirm a single request from the response headers. x-amz-rdma-reply: 200 means the request was served over RDMA, 206 means a ranged or part request was, and 501 means the server declined and returned an S3 error rather than the object. On a successful RDMA transfer the server also sets Content-Length: 0, because the object bytes travelled outside the HTTP response.

For the metric reference, see Metrics version 3 and Validate the RDMA deployment.

Diagnose fabric, adapter, and server problems with Troubleshoot RDMA. The symptoms below appear only on Kubernetes.

Cause: The injected anti-affinity allows one server per node, and the pool requests more servers than there are schedulable RDMA-capable nodes.

Solution: Confirm the cause, then reduce servers or add nodes:

kubectl -n OBJECT-STORE-NAMESPACE describe pod OBJECT-STORE-NAME-pool-0-0

The events report didn't match pod anti-affinity rules.

Cause: The namespace does not permit privileged Pods, or a policy engine blocks privileged, hostNetwork, hostPath, or IPC_LOCK.

Solution: Apply the Pod Security Admission labels from Cluster, and review any third-party policy engine. On OpenShift, bind the privileged SCC as shown in OpenShift.

Cause: The server is binding addresses that do not work.

Solution: Check each layer in order.

1. Confirm the fabric port is ACTIVE, as described in Configure the RDMA fabric.

2. Confirm an IPv4 GID exists for the fabric address. Without a GID there is no RoCE v2.

3. Confirm the labels hold the intended addresses:

   kubectl get nodes -L aistor.min.io/rdma-ip -L aistor.min.io/rdma-ip2
   

4. Confirm the fabric subnet is routable between nodes.

5. With TLS, confirm the certificates carry the fabric addresses as IP subject alternative names. A certificate without them produces repeated remote disconnected messages in the server log and the pools never become ready.

Cause: Fabric mode requires MinIO configuration version 3, and the pinned image predates it.

Solution: Use the RDMA build of AIStor Server, or remove fabric and run the single-NIC topology.

Cause: Fabric addresses bind at process start, and a configuration reload does not rebind adapters.

Solution: The Operator rolls the Pods by changing the aistor.min.io/rdma-config-hash annotation on the Pod template. When that annotation is unchanged, the Operator considers the addresses unchanged. Compare it before and after the edit.

Cause: Inter-node RDMA pins 16 GiB per node by default.

Solution: Reduce MINIO_RDMA_POOL_COUNT or MINIO_RDMA_POOL_BUF_SIZE, or set objectStore.rdma.internode to false. See Locked memory.

• RDMA Acceleration for the concepts and the bare-metal runbook.
• RDMA settings for the full list of server settings.
• Transfer Objects over RDMA for writing an RDMA-aware client.
• Benchmarking for measuring your own deployment.