RDMA Settings

This page documents the environment variables that control RDMA (Remote Direct Memory Access) transfers in MinIO AIStor.

For the deployment procedure these settings belong to, see RDMA acceleration.

These settings require the RDMA build

Every setting on this page is compiled out of the standard AIStor Server binary. On a standard binary the server ignores all of them, including MINIO_RDMA_INTERNODE, and serves every request over TCP without logging anything unusual.

Install the RDMA build first. See Deploy the RDMA server.

Invalid values are ignored silently

None of these settings validate their input at startup. A value that does not parse, falls outside the accepted range, or uses the wrong letter case is discarded and the default is used instead, with no log message and no startup failure. An empty value is treated the same as an unset one.

Confirm each value took effect rather than assuming it did.

All RDMA settings are read once at server startup. Set them in the systemd environment file, /etc/default/minio on a package install, and restart the server to apply a change. There is no mc admin config set equivalent.

Inter-node RDMA

These settings control RDMA transfers of erasure-coded shards between AIStor nodes. They need an RDMA fabric but no GPU.

Enable inter-node RDMA

Set to on to move inter-node shard transfers over RDMA.

Defaults to off. Only the exact string on enables the feature. Any other value leaves it off.

Set the same value on every node in the deployment.

Peer port

The port on which each node listens for RDMA peer connections.

Defaults to 5555.

Open this port between cluster nodes only.

Minimum transfer size

The smallest shard transfer, in bytes, that uses RDMA. Smaller transfers use TCP, because the RDMA setup cost outweighs the benefit.

Defaults to 1048576, which is 1 MiB.

This threshold applies only to the inter-node path. S3 over RDMA has no minimum object size.

Receive-buffer pool

The inter-node path pre-registers a pool of receive buffers with the NIC at startup. A PUT shard that fits in one slab is served from the pool without a per-operation memory registration. A shard larger than a slab, or an arrival while the pool is exhausted, falls back to TCP. That fallback is graceful backpressure, not an error, and it increments minio_system_network_internode_rdma_pool_fallbacks_total.

MINIO_RDMA_POOL_COUNT sets the number of slabs in the pool and defaults to 256.

MINIO_RDMA_POOL_BUF_SIZE sets the size of each slab in bytes and defaults to 67108864, which is 64 MiB. The default is sized for the erasure shard of a typical 64 MiB object.

The product of the two is pinned in physical memory at startup: 16 GiB per node at the defaults.

This memory is locked, so the service needs a locked-memory limit above it. See Raise the locked memory limit.

On a memory-constrained node, lower the count, or lower the slab size and raise the count to compensate. Lowering the slab size below your typical shard size sends those shards to TCP.

Transfer concurrency

The maximum number of in-flight RDMA transfers per node, counting both reads and writes.

Defaults to 512. Set to 0 to remove the bound.

A transfer that cannot get a slot blocks until one frees, rather than failing.

Advanced inter-node tuning

The defaults work without tuning. Change these only to characterize a fabric, or to trade throughput for latency or memory.

Variable Default Description
MINIO_RDMA_INTERNODE_MAX_WRITES_PER_PEER 32 Concurrent in-flight RDMA writes to a single remote node. A coarse outer guard behind the adaptive window below. 32 is the measured optimum on 400GbE; a much lower cap reduces throughput sharply.
MINIO_RDMA_INTERNODE_INFLIGHT_BYTES 1048576 Initial size in bytes of the per-peer adaptive inflight-bytes window, the primary incast control. Roughly one bandwidth-delay product. The window grows additively on each clean completion and halves on a congestion signal.
MINIO_RDMA_INTERNODE_INFLIGHT_BYTES_MIN 262144 Floor of the adaptive window in bytes. The window never shrinks below this, so a peer always makes forward progress.
MINIO_RDMA_INTERNODE_INFLIGHT_BYTES_MAX 4194304 Ceiling of the adaptive window in bytes. Raise it on a confirmed-lossless fabric for more pipelining; lower it if a lossy fabric still reports RETRY_EXC.
MINIO_RDMA_INTERNODE_NUM_DCIS 256 Dynamically Connected initiators per NIC.
MINIO_RDMA_INTERNODE_CQ_DEPTH 256 Completion queue depth per initiator.
MINIO_RDMA_INTERNODE_MAX_WR 256 Maximum outstanding work requests per initiator.
MINIO_RDMA_INTERNODE_GRANT_THRESHOLD 1024 Receive slots that must free before the receiver broadcasts a credit grant.
MINIO_RDMA_INTERNODE_CREDIT_WAIT_MS 60000 Milliseconds a sender blocks waiting for receive credits before reporting a credit timeout. The deadline is deliberately long: the adaptive window already slows senders under congestion, so only a genuinely dead peer should reach it.
MINIO_RDMA_INTERNODE_DATA_POLL_TIMEOUT_MS 5000 Milliseconds to poll for a work request’s completion before treating the path as stuck.
MINIO_RDMA_INTERNODE_MAX_TRANSIENT_RETRIES 2 Retries on a transient transport error such as RETRY_EXC, RNR, or a poll timeout. 0 fails fast.
MINIO_RDMA_INTERNODE_TRAFFIC_CLASS 0 GRH traffic class, which is the DSCP value shifted left by 2. 0 selects automatic detection of the lossless class from the host’s DCB configuration. Set a value only to override, for example 104 for DSCP 26. Accepts 0 through 255.
MINIO_RDMA_INTERNODE_SERVICE_LEVEL 0 RoCE service level, also called PCP. Automatic detection sets it from the detected lossless priority, so override it only on a fabric that trusts PCP, and pair it with MINIO_RDMA_INTERNODE_TRAFFIC_CLASS. Accepts 0 through 7.
Two settings can prevent RDMA from starting

MINIO_RDMA_INTERNODE_SRQ_SIZE and MINIO_RDMA_INTERNODE_INITIAL_CREDITS both default to 32767, which is the maximum shared receive queue depth of a ConnectX-7 adapter.

On an adapter with a smaller max_srq_wr, RDMA initialization fails at startup until you lower MINIO_RDMA_INTERNODE_SRQ_SIZE to match. Read your adapter’s maximum from ibv_devinfo -v before changing either value.

S3 over RDMA (GPU-Direct)

These settings tune RDMA transfers of object data between a client GPU and the server.

S3 over RDMA has no enable setting. The client sends the x-amz-rdma-token header, and the server either serves the request over RDMA or declines it with x-amz-rdma-reply: 501.

Two settings share a name pattern but not a range
MINIO_RDMA_SERVICE_LEVEL accepts 0 through 15, while the inter-node MINIO_RDMA_INTERNODE_SERVICE_LEVEL accepts only 0 through 7. The two paths are configured independently.
Variable Default Description
MINIO_RDMA_CHANNELS 256 Concurrent RDMA channels per node.
MINIO_RDMA_NUM_DCIS 256 Dynamically Connected initiators to allocate. Keep it at or above MINIO_RDMA_CHANNELS to avoid resource exhaustion. Nothing enforces this; a lower value produces no warning.
MINIO_RDMA_CQ_DEPTH 4096 Completion queue depth for pending operations.
MINIO_RDMA_TIMEOUT 14 Local ACK timeout, expressed as an exponent rather than a duration. The timeout is 4.096 microseconds multiplied by 2 to the power of this value, so 14 is about 67 ms per attempt and 16 is about 268 ms. Lower values detect failure faster.
MINIO_RDMA_RETRY_COUNT 7 Transport retries before the operation fails with IBV_WC_RETRY_EXC_ERR. The total budget is (1 + retry count) multiplied by the per-attempt timeout, so the defaults give about 537 ms. Accepts 0 through 7.
MINIO_RDMA_MAX_SGE 16 Scatter-gather entries per operation, which sets how many buffers a single RDMA operation can batch.
MINIO_RDMA_DELAY_MODE batch When polling delays apply. Accepts none, batch, entry, or adaptive, or the equivalent integers 0 through 3. The names are matched in lowercase only, so BATCH is discarded and the default is kept.
MINIO_RDMA_DELAY_INTERVAL 500 Polling delay in nanoseconds. Lower values reduce latency and raise CPU use.
MINIO_RDMA_POLL_INTERVAL_US 50 Completion polling interval in microseconds. 10 to 20 suits a 400G fabric; 50 to 100 suits 100G.
MINIO_RDMA_QP_RESET_ON_FAILURE on Reset queue pairs automatically after an error. Only the exact string on keeps this enabled; any other value disables it.
MINIO_RDMA_SERVICE_LEVEL 0 InfiniBand service level for QoS. Accepts 0 through 15.
MINIO_RDMA_TRAFFIC_CLASS 0 DSCP or ToS byte for a lossless fabric. 0 means best effort. Unlike the inter-node setting, 0 here does not trigger automatic detection.
MINIO_RDMA_HOP_LIMIT 4 Packet lifetime, the RDMA analogue of IP TTL. Raise it for multi-hop fabrics.
MINIO_RDMA_PKEY_INDEX 0 Partition key index. Relevant only on a partitioned InfiniBand fabric.
MINIO_RDMA_MAX_RD_ATOMIC 1 Outstanding RDMA read and atomic operations per queue pair. Must be at least 1; a value of 0 is ignored because it would disable RDMA reads on the queue pair.

Bandwidth limits

If the deployment runs with a server bandwidth cap, that cap paces S3-over-RDMA client transfers in both directions, the same as it paces HTTP transfers. The server logs RDMA: GPU-Direct bandwidth limited to <n> B/s per direction at startup when a cap is in effect.

The cap does not apply to inter-node RDMA. Shard traffic between nodes is bounded by MINIO_RDMA_CONCURRENCY and the adaptive inflight-bytes window instead.