Troubleshoot RDMA

Diagnose RDMA problems by symptom.

Each entry names what you observe, what causes it, and how to confirm and fix it. If you have not yet worked through Validate the RDMA deployment, start there: it isolates which layer is at fault, which usually identifies the entry below to read.

Symptom, inter-node RDMA. The minio_system_network_internode_rdma_* metrics stay at zero and shard transfers use TCP. The server behaves normally otherwise, because inter-node RDMA falls back to TCP transparently.

Symptom, S3 over RDMA. The minio_api_rdma_* metrics stay at zero and clients see errors, not slower transfers. S3 over RDMA does not fall back on the server side: a request carrying x-amz-rdma-token that the server cannot serve over RDMA returns an S3 error response, usually with x-amz-rdma-reply: 501. A client that does not retry without the header reports a failure.

Both share the same causes. Work through them in order.

This is the most common cause, and the one that produces no error message anywhere. Every RDMA setting is compiled out of the standard binary, so MINIO_RDMA_INTERNODE=on is silently ignored.

/usr/local/bin/minio.rdma --version

The RDMA build prints Features: GPU-Direct RDMA in its version banner. If that line is absent, install the RDMA package. See Deploy the RDMA server.

Installing the RDMA package does not repoint the service. The packaged unit runs /usr/local/bin/minio, which is the standard binary.

systemctl show minio --property=ExecStart

The path must end in minio.rdma. If it does not, add the systemd drop-in described in Point the service at the RDMA binary.

This also happens after a working deployment, if someone repointed the /usr/local/bin/minio symlink instead of using a drop-in: the next package upgrade restores the symlink and the node reverts to TCP.

ibv_devinfo

Without a device reporting state: PORT_ACTIVE (4), inter-node shard transfers fall back to TCP, and S3 requests carrying x-amz-rdma-token are declined with x-amz-rdma-reply: 501. Load the kernel modules and check the link. See Confirm the host sees an RDMA device.

journalctl -u minio | grep -i rdma

The absence of RDMA: internode initialized with ... means the inter-node path never came up. The line RDMA: GPU-Direct S3 server disabled (cuObjServer init failed) means the GPU-Direct path failed; the server runs but declines every RDMA request.

For the full list of startup lines and what each means, see Confirm initialization succeeded.

sudo systemctl show minio --property=Environment
grep MINIO_RDMA /etc/default/minio

RDMA settings are read once at startup from the systemd environment file. A setting added to /etc/minio/config.env after startup has no effect.

Symptom. RDMA: inbound receive pool ready never appears, and the log carries an out-of-memory or registration error. The server starts anyway: an RDMA initialization failure is logged, not fatal.

This pool belongs to the inter-node path, so the effect is that shard transfers fall back to TCP. The GPU-Direct path initializes separately and can still be serving S3 over RDMA. Check it with the log lines in Confirm initialization succeeded.

Cause. The locked memory limit is below what RDMA needs. A distributed node at default settings registers roughly 40 GiB, and the packaged systemd unit sets no LimitMEMLOCK, so the service inherits the systemd default of 8 MiB.

This is not fixed by /etc/security/limits.conf. That file is read by PAM during login, and a systemd-managed service never goes through PAM.

Check the limit the service actually received:

systemctl show minio --property=LimitMEMLOCK

Set it in the drop-in. See Raise the locked memory limit.

Also confirm the service account can open the RDMA devices:

ls -la /dev/infiniband/

Symptom. RDMA initializes on some nodes and fails at startup on others, with otherwise identical configuration.

Cause. MINIO_RDMA_INTERNODE_SRQ_SIZE defaults to 32767, the maximum shared receive queue depth of a ConnectX-7 adapter. On an adapter with a smaller max_srq_wr, requesting more slots than the hardware supports fails initialization.

Read your adapter’s maximum:

ibv_devinfo -v | grep max_srq_wr

Lower MINIO_RDMA_INTERNODE_SRQ_SIZE to at most that value and restart.

Symptom. S3-over-RDMA transfers into GPU memory fail with a remote operation error, completion status 11, for example RDMA PUT: remote operation error (code: 11). Frequently only the GPU physically closest to the NIC fails while other GPUs succeed.

Cause. PCIe ACS redirect is blocking GPU-to-NIC peer-to-peer DMA, so the NIC cannot reach the target GPU’s memory.

Fix. Clear ACS redirect on the bridges in the GPU and NIC data path. See Clear PCIe ACS redirect.

Confirm the fix took effect. The bridges above the GPU and NIC should report ReqRedir- and CmpltRedir-:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

If the fix worked once and stopped working after a reboot, the systemd one-shot did not run. Firmware resets ACS state on every boot. See Confirm the fix took effect.

Symptom. Inter-node RDMA works, but throughput falls far below line rate during writes, sometimes to a few Gb/s per node. Errors climb.

Cause. The fabric is not lossless. The all-to-all erasure-write pattern causes incast; the switch overruns its egress buffer and drops packets, which surfaces as IBV_WC_RETRY_EXC_ERR.

Confirm by reading the counters under load:

ethtool -S <dev> | grep -iE 'prio3_pause'
grep -r . /sys/class/infiniband/<rdma_dev>/ports/1/hw_counters/ | \
    grep -iE 'packet_seq_err|local_ack_timeout_err'

Pause counters flat while packet_seq_err or local_ack_timeout_err climb means the switch is dropping rather than pausing.

Fix. Configure PFC and DCQCN end to end, on every host and every switch in the path. See Make the RoCE fabric lossless.

Host-only configuration does nothing if the switch drops.

If the fabric is confirmed lossless and throughput is still short, check minio_system_network_internode_rdma_nic_credit_waits_total and ..._nic_writes_throttled_total. Sustained growth in either means senders are being held back, and MINIO_RDMA_INTERNODE_INFLIGHT_BYTES_MAX may be worth raising on a confirmed-lossless fabric.

Symptom. No errors, no drops, but per-node throughput plateaus at a fraction of line rate.

Cause. On a 400G fabric, the default ECN and RED marking thresholds on NVIDIA Spectrum switches are tuned for low-speed links and over-throttle DCQCN. They mark long before the pipe is full, so a fabric with no errors and no drops still plateaus well below what it can carry.

Fix. Raise the thresholds. See Configure the switch.

Symptom. RDMA works, but a portion of inter-node traffic still uses TCP.

Check the fallback counter:

curl -s http://localhost:9000/minio/metrics/v3/system/network/internode/rdma | \
    grep pool_fallbacks_total

A rising minio_system_network_internode_rdma_pool_fallbacks_total is graceful backpressure, not an error. It means a PUT shard arrived while the receive-buffer pool was busy, or was larger than one pool slab, and the transfer used TCP instead.

Two causes, with different fixes:

• Shards larger than a slab. MINIO_RDMA_POOL_BUF_SIZE defaults to 64 MiB, sized for the erasure shard of a typical 64 MiB object. A workload of much larger objects produces larger shards. Raise the slab size, and account for the extra locked memory.
• Pool exhaustion under concurrency. Raise MINIO_RDMA_POOL_COUNT, again accounting for locked memory.

Transfers below MINIO_RDMA_MIN_SIZE, 1 MiB by default, always use TCP and do not increment this counter.

Symptom. Throughput is lower than it was, and the log carries:

RDMA: sender buffer registration failed (falling back to per-write reg): <error>

Cause. The node could not pre-register its sender buffers, so it registers memory on every write instead of reusing pre-registered regions. Transfers still succeed, at a cost.

Locked memory pressure is the usual reason. Check LimitMEMLOCK and the node’s memory headroom, and read the error text in the log line for the specific failure.

Symptom. RDMA operations time out.

Confirm basic reachability and that the peer port is open:

ping <peer-node>
nc -zv <peer-node> 5555

If the port is closed, see Open the peer port.

On a congested fabric, the ACK timeout may be too aggressive. MINIO_RDMA_TIMEOUT is an exponent, not a duration: the timeout is 4.096 microseconds multiplied by 2 to the power of the value. The default of 14 gives about 67 ms per attempt, and 16 gives about 268 ms.

Raise it only after confirming the fabric is lossless. A timeout increase hides a dropping fabric rather than fixing it.

Symptom. Inter-node RDMA transfers fail on a multi-rail host whose rails share a layer-2 broadcast domain, while S3 over RDMA to clients works normally. The server logs both of:

RDMA: WriteBuffer to client failed: context deadline exceeded
RDMA: WriteBuffer to client failed: RDMA WRITE failed at offset 0:
  p2p_rdma_write_one failed: transport error: work completion error:
  IBV_WC_RETRY_EXC_ERR (status=12, vendor_err=129)

IBV_WC_RETRY_EXC_ERR means the responder never acknowledged, so the sender exhausted its transport retries. On a multi-rail host the usual cause is ARP resolving a peer’s rail address to the wrong rail’s MAC. See Multi-rail hosts sharing a broadcast domain need interface-scoped ARP for why TCP keeps working while RoCE fails.

Two properties make this easy to misdiagnose. Every transfer over an affected rail fails, but a rail that happened to resolve correctly keeps working, so the loss looks partial. Single-NIC peers are unaffected, because one MAC legitimately serves all of their addresses, so server-to-client traffic looks healthy while server-to-server traffic does not.

Test one rail pair directly, outside AIStor, to separate the selected path from AIStor’s behavior. This exercises only the devices, GIDs and path you name, so it does not by itself clear the whole fabric. Start the receiver first. Use matching test parameters on both ends, but give each host its own device and GID index: those name the local port and address, and the numeric index routinely differs between hosts even when both identify a compatible RoCE v2 path. Only the sender takes an address.

On the receiver:

ib_write_bw -d "$RECV_DEV" -x "$RECV_GID" -F -D 5

On the sender, once the receiver is listening:

ib_write_bw -d "$SEND_DEV" -x "$SEND_GID" -F -D 5 "$RECV_IP"

Find each host’s RoCE v2 GID index for the address under test. The table length is device- and driver-dependent, and the driver may reserve entries, so enumerate it rather than assuming a low index. The loop below scans every sysfs GID entry and prints the populated ones:

dev=mlx5_0 # the local RDMA device
port=1     # the port ib_write_bw uses by default

for gid in /sys/class/infiniband/$dev/ports/$port/gids/*; do
    value=$(cat "$gid" 2>/dev/null)
    case "$value" in 0000:0000:0000:0000:0000:0000:0000:0000 | "") continue ;; esac
    index=$(basename "$gid")
    type=$(cat /sys/class/infiniband/$dev/ports/$port/gid_attrs/types/$index 2>/dev/null)
    ndev=$(cat /sys/class/infiniband/$dev/ports/$port/gid_attrs/ndevs/$index 2>/dev/null)
    printf '%-3s %-12s %-40s %s\n' "$index" "$type" "$value" "$ndev"
done

Pick the RoCE v2 row whose value ends in the IPv4 address under test. An IPv4 GID is ::ffff: followed by the address in hex, so 15.15.15.60 appears as ...:ffff:0f0f:0f3c.

A run that prints the results header and then transfers nothing is consistent with this failure rather than proof of it, because a wrong device or GID index prevents transfer the same way. Before attributing it, confirm the GID index really names the address under test and that ip neigh maps that address to the owning interface’s MAC. A single-NIC peer running at line rate validates only the path actually tested, so it narrows the fault to the multi-rail pairing rather than clearing the whole fabric.

Then compare the learned MACs against the interfaces that own them, as described in the fabric guide.

After applying the fix, restart AIStor on every node. Cached address handles keep using the stale MAC for the life of the process, so the fix looks ineffective until the service restarts.

For a stuck inter-node transfer that none of the above explains, enable verbose completion logging:

MINIO_RDMA_DEBUG=on

Add it to /etc/default/minio, restart the node, and read the output:

journalctl -u minio | grep 'RDMA-DEBUG:'

Only the exact string on enables it. Any other value leaves it off.

This switch instruments the inter-node path only. It produces no output for an S3-over-RDMA transfer.

When opening a case, include the output of mc support diag ALIAS along with ibv_devinfo, journalctl -u minio | grep -i rdma, and the two RDMA metric groups. See SUBNET.