Validate the RDMA deployment

Prove that each layer of the RDMA stack works before you rely on it.

RDMA fails quietly. A node with the wrong binary, an inactive port, or a lossy fabric starts normally and serves every request over TCP. Work up the layers in order, because a failure at a lower layer makes every check above it meaningless.

ibv_devinfo

Each usable port reports state: PORT_ACTIVE (4). Any other state, or no device at all, means the server falls back to HTTP for every transfer.

List the RDMA links and their netdev bindings:

rdma link show

The bridges above the GPU and the NIC must report ACS redirect cleared:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

Look for ReqRedir- and CmpltRedir-. If either shows +, work through Clear PCIe ACS redirect.

Confirm the one-shot ran on the current boot rather than trusting systemctl is-enabled:

sudo journalctl -u disable-pcie-acs.service -b

Put the fabric under all-to-all load, then read the counters. The perftest package, available as perftest on Debian and Ubuntu and perftest on RHEL derivatives, provides ib_write_bw for a point-to-point baseline:

# On the receiving node
ib_write_bw -d mlx5_0

# On the sending node
ib_write_bw -d mlx5_0 <receiver-ip>

A point-to-point test does not reproduce incast, which is the pattern that exposes a lossy fabric. For an all-to-all load, run a write benchmark against the deployment itself with warp and watch the counters during it. See Benchmarking.

While the load runs, confirm the fabric pauses rather than drops:

ethtool -S <dev> | grep -iE 'prio3_pause'
grep -r . /sys/class/infiniband/<rdma_dev>/ports/1/hw_counters/ | \
    grep -iE 'packet_seq_err|local_ack_timeout_err'

Pause counters rising is the good signal. packet_seq_err and local_ack_timeout_err must stay flat. See Confirm losslessness is working.

/usr/local/bin/minio.rdma --version

The version banner includes this line only on the RDMA build:

Features: GPU-Direct RDMA

This proves the build and nothing else. It says nothing about the fabric, the adapter, or whether initialization succeeded.

Confirm the running service is that binary:

systemctl show minio --property=ExecStart

ldd /usr/local/bin/minio.rdma | grep -E "ibverbs|rdmacm|s3rdma|p2p_rdma"

Every line must resolve to a path. libs3rdma.so.0 and libp2p_rdma.so.0 come from /usr/lib/minio.

Inter-node RDMA logs a single definitive success line:

journalctl -u minio | grep "RDMA: internode initialized"

RDMA: internode initialized with 15 remote peer(s), 2 local NIC(s), primary listener 10.0.0.1:5555

The local NIC count in that line is also how you confirm how many rails the server detected.

Two lines precede it and are worth reading:

For the GPU-Direct path:

journalctl -u minio | grep "GPU-Direct"

One line signals a degraded but working state:

RDMA: sender buffer registration failed (falling back to per-write reg): <error>

The node still transfers over RDMA, but registers memory on every write instead of reusing pre-registered buffers. Throughput suffers. Treat it as a fault to investigate.

Scrape the two RDMA metric groups:

curl http://localhost:9000/minio/metrics/v3/api/rdma
curl http://localhost:9000/minio/metrics/v3/system/network/internode/rdma

A rising pool_fallbacks_total is not an error. It means a PUT shard arrived while the receive-buffer pool was busy, or was larger than one slab, and the transfer used TCP instead. Sustained growth means the pool is undersized for the workload. See Receive-buffer pool.

For the full metric list, see Metrics version 3.

A client proves RDMA served its request by reading the response headers.

On a successful RDMA transfer the server also sets Content-Length: 0, because the object bytes travelled outside the HTTP response.

Confirm from the server side as well. Read the counter matching the transfer direction before and after, and check that it increased: minio_api_rdma_read_ops_total for a GET, minio_api_rdma_write_ops_total for a PUT or part upload.

See Troubleshoot RDMA.