Troubleshoot RDMA

Diagnose RDMA problems by symptom.

Each entry names what you observe, what causes it, and how to confirm and fix it. If you have not yet worked through Validate the RDMA deployment, start there: it isolates which layer is at fault, which usually identifies the entry below to read.

Symptom, inter-node RDMA. The minio_system_network_internode_rdma_* metrics stay at zero and shard transfers use TCP. The server behaves normally otherwise, because inter-node RDMA falls back to TCP transparently.

Symptom, S3 over RDMA. The minio_api_rdma_* metrics stay at zero and clients see errors, not slower transfers. S3 over RDMA does not fall back on the server side: a request carrying x-amz-rdma-token that the server cannot serve over RDMA returns an S3 error response, usually with x-amz-rdma-reply: 501. A client that does not retry without the header reports a failure.

Both share the same causes. Work through them in order.

This is the most common cause, and the one that produces no error message anywhere. Every RDMA setting is compiled out of the standard binary, so MINIO_RDMA_INTERNODE=on is silently ignored.

/usr/local/bin/minio.rdma --version

The RDMA build prints Features: GPU-Direct RDMA in its version banner. If that line is absent, install the RDMA package. See Deploy the RDMA server.

Installing the RDMA package does not repoint the service. The packaged unit runs /usr/local/bin/minio, which is the standard binary.

systemctl show minio --property=ExecStart

The path must end in minio.rdma. If it does not, add the systemd drop-in described in Point the service at the RDMA binary.

This also happens after a working deployment, if someone repointed the /usr/local/bin/minio symlink instead of using a drop-in: the next package upgrade restores the symlink and the node reverts to TCP.

ibv_devinfo

Without a device reporting state: PORT_ACTIVE (4), inter-node shard transfers fall back to TCP, and S3 requests carrying x-amz-rdma-token are declined with x-amz-rdma-reply: 501. Load the kernel modules and check the link. See Confirm the host sees an RDMA device.

journalctl -u minio | grep -i rdma

The absence of RDMA: internode initialized with ... means the inter-node path never came up. The line RDMA: GPU-Direct S3 server disabled (cuObjServer init failed) means the GPU-Direct path failed; the server runs but declines every RDMA request.

For the full list of startup lines and what each means, see Confirm initialization succeeded.

sudo systemctl show minio --property=Environment
grep MINIO_RDMA /etc/default/minio

RDMA settings are read once at startup from the systemd environment file. A setting added to /etc/minio/config.env after startup has no effect.

Symptom. RDMA: inbound receive pool ready never appears, and the log carries an out-of-memory or registration error. The server starts anyway: an RDMA initialization failure is logged, not fatal.

This pool belongs to the inter-node path, so the effect is that shard transfers fall back to TCP. The GPU-Direct path initializes separately and can still be serving S3 over RDMA. Check it with the log lines in Confirm initialization succeeded.

Cause. The locked memory limit is below what RDMA needs. A distributed node at default settings registers roughly 40 GiB, and the packaged systemd unit sets no LimitMEMLOCK, so the service inherits the systemd default of 8 MiB.

This is not fixed by /etc/security/limits.conf. That file is read by PAM during login, and a systemd-managed service never goes through PAM.

Check the limit the service actually received:

systemctl show minio --property=LimitMEMLOCK

Set it in the drop-in. See Raise the locked memory limit.

Also confirm the service account can open the RDMA devices:

ls -la /dev/infiniband/

Symptom. RDMA initializes on some nodes and fails at startup on others, with otherwise identical configuration.

Cause. MINIO_RDMA_INTERNODE_SRQ_SIZE defaults to 32767, the maximum shared receive queue depth of a ConnectX-7 adapter. On an adapter with a smaller max_srq_wr, requesting more slots than the hardware supports fails initialization.

Read your adapter’s maximum:

ibv_devinfo -v | grep max_srq_wr

Lower MINIO_RDMA_INTERNODE_SRQ_SIZE to at most that value and restart.

Symptom. S3-over-RDMA transfers into GPU memory fail with a remote operation error, completion status 11, for example RDMA PUT: remote operation error (code: 11). Frequently only the GPU physically closest to the NIC fails while other GPUs succeed.

Cause. PCIe ACS redirect is blocking GPU-to-NIC peer-to-peer DMA, so the NIC cannot reach the target GPU’s memory.

Fix. Clear ACS redirect on the bridges in the GPU and NIC data path. See Clear PCIe ACS redirect.

Confirm the fix took effect. The bridges above the GPU and NIC should report ReqRedir- and CmpltRedir-:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

If the fix worked once and stopped working after a reboot, the systemd one-shot did not run. Firmware resets ACS state on every boot. See Confirm the fix took effect.

Symptom. Inter-node RDMA works, but throughput falls far below line rate during writes, sometimes to a few Gb/s per node. Errors climb.

Cause. The fabric is not lossless. The all-to-all erasure-write pattern causes incast; the switch overruns its egress buffer and drops packets, which surfaces as IBV_WC_RETRY_EXC_ERR.

Confirm by reading the counters under load:

ethtool -S <dev> | grep -iE 'prio3_pause'
grep -r . /sys/class/infiniband/<rdma_dev>/ports/1/hw_counters/ | \
    grep -iE 'packet_seq_err|local_ack_timeout_err'

Pause counters flat while packet_seq_err or local_ack_timeout_err climb means the switch is dropping rather than pausing.

Fix. Configure PFC and DCQCN end to end, on every host and every switch in the path. See Make the RoCE fabric lossless.

Host-only configuration does nothing if the switch drops.

If the fabric is confirmed lossless and throughput is still short, check minio_system_network_internode_rdma_nic_credit_waits_total and ..._nic_writes_throttled_total. Sustained growth in either means senders are being held back, and MINIO_RDMA_INTERNODE_INFLIGHT_BYTES_MAX may be worth raising on a confirmed-lossless fabric.

Symptom. No errors, no drops, but per-node throughput plateaus at a fraction of line rate.

Cause. On a 400G fabric, the default ECN and RED marking thresholds on NVIDIA Spectrum switches are tuned for low-speed links and over-throttle DCQCN. MinIO measured the defaults capping a clean fabric at about 104 Gb/s per node.

Fix. Raise the thresholds. See Configure the switch.

Symptom. RDMA works, but a portion of inter-node traffic still uses TCP.

Check the fallback counter:

curl -s http://localhost:9000/minio/metrics/v3/system/network/internode/rdma | \
    grep pool_fallbacks_total

A rising minio_system_network_internode_rdma_pool_fallbacks_total is graceful backpressure, not an error. It means a PUT shard arrived while the receive-buffer pool was busy, or was larger than one pool slab, and the transfer used TCP instead.

Two causes, with different fixes:

• Shards larger than a slab. MINIO_RDMA_POOL_BUF_SIZE defaults to 64 MiB, sized for the erasure shard of a typical 64 MiB object. A workload of much larger objects produces larger shards. Raise the slab size, and account for the extra locked memory.
• Pool exhaustion under concurrency. Raise MINIO_RDMA_POOL_COUNT, again accounting for locked memory.

Transfers below MINIO_RDMA_MIN_SIZE, 1 MiB by default, always use TCP and do not increment this counter.

Symptom. Throughput is lower than it was, and the log carries:

RDMA: sender buffer registration failed (falling back to per-write reg): <error>

Cause. The node could not pre-register its sender buffers, so it registers memory on every write instead of reusing pre-registered regions. Transfers still succeed, at a cost.

Locked memory pressure is the usual reason. Check LimitMEMLOCK and the node’s memory headroom, and read the error text in the log line for the specific failure.

Symptom. RDMA operations time out.

Confirm basic reachability and that the peer port is open:

ping <peer-node>
nc -zv <peer-node> 5555

If the port is closed, see Open the peer port.

On a congested fabric, the ACK timeout may be too aggressive. MINIO_RDMA_TIMEOUT is an exponent, not a duration: the timeout is 4.096 microseconds multiplied by 2 to the power of the value. The default of 14 gives about 67 ms per attempt, and 16 gives about 268 ms.

Raise it only after confirming the fabric is lossless. A timeout increase hides a dropping fabric rather than fixing it.

For a stuck inter-node transfer that none of the above explains, enable verbose completion logging:

MINIO_RDMA_DEBUG=on

Add it to /etc/default/minio, restart the node, and read the output:

journalctl -u minio | grep 'RDMA-DEBUG:'

Only the exact string on enables it. Any other value leaves it off.

This switch instruments the inter-node path only. It produces no output for an S3-over-RDMA transfer.

When opening a case, include the output of mc support diag ALIAS along with ibv_devinfo, journalctl -u minio | grep -i rdma, and the two RDMA metric groups. See SUBNET.