RDMA Acceleration

MinIO AIStor can move object data over RDMA (Remote Direct Memory Access) instead of TCP. RDMA writes directly into the memory of the remote machine, so the payload never passes through the kernel network stack or an intermediate copy.

Version added

RDMA has shipped in AIStor edge builds since EDGE.2026-06-01T02-39-13Z, where it was hardened across several releases. RELEASE.2026-08-07T18-34-35Z promotes it to the stable release channel, and is the first stable release to publish RDMA binaries, packages, and container images.

Deployments that ran RDMA from an edge build should move to the stable RDMA artifacts described in Deploy the RDMA server.

RDMA is not part of the standard AIStor Server binary. It ships as a separate build that you download and install in place of the standard one. See Deploy the RDMA server.

Every RDMA setting is compiled out of the standard binary. On a standard binary, MINIO_RDMA_INTERNODE=on is a silent no-op: the server starts, logs nothing unusual, and serves every request over TCP. Install the RDMA build first.
A GetObject request travels over HTTP through the kernel network stack on both hosts, carrying only headers, while the object payload travels separately across the RDMA fabric from the drives directly into client GPU memory.

A GetObject served over RDMA. The request and response are ordinary signed S3 traffic; the object itself never enters the kernel network stack.

The two RDMA paths

AIStor uses RDMA on two independent paths. They have different hardware prerequisites and different settings, and you can enable either one on its own. Decide which path you are deploying before you start, because several setup steps apply to only one of them.

Path What it moves What it needs How you enable it
S3 over RDMA (GPU-Direct) Object data between a client GPU and the server An NVIDIA GPU on the client, GPU-to-NIC peer-to-peer DMA, and a client that sends the x-amz-rdma-token header No setting. The client sends the header and the server either serves the request over RDMA or declines it.
Inter-node RDMA Erasure-coded shards between AIStor nodes An RDMA fabric. No GPU. Set MINIO_RDMA_INTERNODE=on on every node.

Three S3 operations have an RDMA path: GetObject, PutObject, and UploadPart. Every other operation uses HTTP regardless of the header.

Pages in this section mark any section that applies to only one path.

What to expect

Inter-node RDMA is a throughput and CPU-efficiency change on the write path. On a 16-node cluster with 400G ConnectX NICs and a lossless RoCE fabric, MinIO measured 367 Gb/s per node in an all-to-all erasure-write test, or about 92% of line rate.

That result depends on the fabric being lossless. The same test on the same NICs behind a switch without Priority Flow Control (PFC) collapses to a few Gb/s per node, because the all-to-all write pattern causes incast and the switch drops packets. Configuring losslessness is the single most important step for inter-node RDMA, and it is why Configure the RDMA fabric comes before the install step in this runbook.

S3 over RDMA removes the copy between server storage and client GPU memory. It benefits GPU training and inference pipelines that read large objects into device memory.

Requirements

RDMA requires Linux on amd64 or arm64. Windows and macOS are not supported.

The RDMA build is separate from the FIPS build. A deployment can use one or the other, not both.

Bare-metal deployments only

RDMA is supported on bare-metal Linux deployments.

The MinIO AIStor Operator does not support RDMA in this release. It has no field for RDMA device resources or locked-memory limits, and its upgrade path does not recognize an RDMA image tag. Do not deploy RDMA on Kubernetes.

Inter-node shards below MINIO_RDMA_MIN_SIZE, 1 MiB by default, transfer over TCP. The RDMA setup cost is not worthwhile below that size. This threshold applies only to the inter-node path. S3 over RDMA has no minimum object size.

When the server cannot serve an S3 request over RDMA, it returns x-amz-rdma-reply: 501 with an S3 error response. It does not silently serve the object over HTTP instead. Retrying without the x-amz-rdma-token header is the client’s responsibility.

The MinIO SDKs for Go, C++, Rust, and Python support S3 over RDMA, each using NVIDIA cuObjClient to move the payload between GPU memory and the network adapter. See Transfer Objects over RDMA.

Run the runbook in order

Work through these pages in sequence. Each one ends with a check you can run before moving on, and skipping ahead tends to produce a deployment that silently falls back to TCP.

  1. Configure the RDMA fabric — hardware, switch, host NIC, and PCIe settings.
  2. Deploy the RDMA server — install the RDMA package and point the service at it.
  3. Validate the RDMA deployment — prove each layer carries RDMA traffic.
  4. Troubleshoot RDMA — symptoms, causes, and fixes.

For the full list of settings, see RDMA settings.