RDMA Acceleration

MinIO AIStor can move object data over RDMA (Remote Direct Memory Access) instead of TCP. RDMA writes directly into the memory of the remote machine, so the payload never passes through the kernel network stack or an intermediate copy.

Version added

RDMA has shipped in AIStor edge builds since EDGE.2026-06-01T02-39-13Z, where it was hardened across several releases. RELEASE.2026-08-07T18-34-35Z promotes it to the stable release channel, and is the first stable release to publish RDMA binaries, packages, and container images.

Deployments that ran RDMA from an edge build should move to the stable RDMA artifacts described in Deploy the RDMA server.

RDMA is not part of the standard AIStor Server binary. It ships as a separate build that you download and install in place of the standard one. See Deploy the RDMA server.

Every RDMA setting is compiled out of the standard binary. On a standard binary, MINIO_RDMA_INTERNODE=on is a silent no-op: the server starts, logs nothing unusual, and serves every request over TCP. Install the RDMA build first.
GetObject over RDMA A GetObject request travels over HTTP through the kernel network stack on both hosts, carrying only headers. The object payload returns on a separate path: from the drives, through the server network adapter, across the RDMA fabric, through the client adapter, and directly into GPU memory, never entering the kernel network stack. CLIENT HOST AISTOR SERVER · RDMA BUILD Application issues GetObject S3 handler verifies the signature Kernel stack headers only Kernel stack headers only HTTP over TCP GPU memory RDMA NIC RDMA NIC Drives object payload RDMA FABRIC InfiniBand or RoCE v2 Data path The object itself. Read straight into GPU memory over PCIe, never copied through the kernel network stack. Control path An ordinary signed S3 request and response, carrying only headers. On the wire request  x-amz-rdma-token reply    x-amz-rdma-reply: 200 The response body is empty — a successful transfer replies Content-Length: 0.
A GetObject served over RDMA. The request and response are ordinary signed S3 traffic; the object itself never enters the kernel network stack.

The two RDMA paths

AIStor uses RDMA on two independent paths. They have different hardware prerequisites and different settings, and you can enable either one on its own. Decide which path you are deploying before you start, because several setup steps apply to only one of them.

Path What it moves What it needs How you enable it
S3 over RDMA Object data between an application and the server A client that sends the x-amz-rdma-token header. Reading into GPU memory also needs an NVIDIA GPU with GPU-to-NIC peer-to-peer DMA. No setting. The client sends the header and the server either serves the request over RDMA or declines it.
Inter-node RDMA Erasure-coded shards between AIStor nodes An RDMA fabric. No GPU. Set MINIO_RDMA_INTERNODE=on on every node.

Three S3 operations have an RDMA path: GetObject, PutObject, and UploadPart. Every other operation uses HTTP regardless of the header.

Pages in this section mark any section that applies to only one path.

What to expect

S3 over RDMA serves the object path of an AI pipeline. Training and inference jobs read large objects continuously and write checkpoints in bursts. At that rate the copy between the server and the application’s memory limits them, rather than the storage. This is the path NVIDIA calls GPU-Direct when the target buffer is in device memory. RDMA removes that copy and delivers the object straight into the buffer the application registered.

Where that buffer lives does not change the deployment. A GPU training node registers device memory; a CPU inference node registers host memory. Both take the same path at the same speed, so a fleet can mix them without a second configuration.

Inter-node RDMA is a separate change, on the write path between nodes. It moves erasure-coded shards over the fabric instead of TCP, which raises write throughput and lowers CPU use on every node.

What you get from it depends entirely on the fabric being lossless. Without Priority Flow Control (PFC), the all-to-all write pattern causes incast, the switch drops packets, and inter-node RDMA performs worse than the TCP path it replaced.

Configuring losslessness is the single most important step for inter-node RDMA. It is why Configure the RDMA fabric comes before the install step in this runbook.

Measure your own deployment rather than working from published figures. See benchmarking.

Requirements

RDMA requires Linux on amd64 or arm64. Windows and macOS are not supported.

The RDMA build is separate from the FIPS build. A deployment can use one or the other, not both.

Bare-metal deployments only

RDMA is supported on bare-metal Linux deployments.

The MinIO AIStor Operator does not support RDMA in this release. It has no field for RDMA device resources or locked-memory limits, and its upgrade path does not recognize an RDMA image tag. Do not deploy RDMA on Kubernetes.

Inter-node shards below MINIO_RDMA_MIN_SIZE, 1 MiB by default, transfer over TCP. The RDMA setup cost is not worthwhile below that size. This threshold applies only to the inter-node path. S3 over RDMA has no minimum object size.

When the server cannot serve an S3 request over RDMA, it returns x-amz-rdma-reply: 501 with an S3 error response. It does not silently serve the object over HTTP instead. Retrying without the x-amz-rdma-token header is the client’s responsibility.

The MinIO SDKs for Go, C++, Rust, and Python support S3 over RDMA, each using NVIDIA cuObjClient to move the payload between GPU memory and the network adapter. See Transfer Objects over RDMA.

Run the runbook in order

Work through these pages in sequence. Each one ends with a check you can run before moving on, and skipping ahead tends to produce a deployment that silently falls back to TCP.

  1. Configure the RDMA fabric — hardware, switch, host NIC, and PCIe settings.
  2. Deploy the RDMA server — install the RDMA package and point the service at it.
  3. Validate the RDMA deployment — prove each layer carries RDMA traffic.
  4. Troubleshoot RDMA — symptoms, causes, and fixes.

For the full list of settings, see RDMA settings.