Configure the RDMA fabric

Prepare the hardware, the switches, and the host operating system before you install the RDMA build of AIStor Server. An AIStor node with an unconfigured fabric starts normally and serves every request over TCP, so a missed step here shows up as “RDMA does nothing” rather than as an error.

Work through the sections in order. Each section names which of the two RDMA paths it applies to.

Both RDMA paths need the following:

On a RoCE fabric, inter-node RDMA additionally requires that the fabric be lossless end to end. See Make the RoCE fabric lossless.

Run this before anything else. If ibv_devinfo reports no device, or the port state is not PORT_ACTIVE, fix the fabric before you continue.

ibv_devinfo

The output lists each adapter and its port state:

hca_id: mlx5_0
  transport:      InfiniBand (0)
  fw_ver:         20.31.1014
  phys_port_cnt:  1
    port:   1
      state:        PORT_ACTIVE (4)
      max_mtu:      4096 (5)
      active_mtu:   4096 (5)

Check the link state and the fabric view as well:

ibstat
ibstatus

Load the RDMA modules on every node:

sudo modprobe ib_core
sudo modprobe ib_uverbs
sudo modprobe rdma_cm
sudo modprobe mlx5_core

Confirm they loaded:

lsmod | grep -E "ib_|rdma_|mlx"

To load them at every boot, create /etc/modules-load.d/rdma.conf:

ib_core
ib_uverbs
rdma_cm
mlx5_core

GPU-Direct RDMA moves the object payload directly between GPU memory and the RDMA NIC over PCIe peer-to-peer DMA. Many servers block this silently through PCIe Access Control Services (ACS) redirect.

When a GPU and the NIC sit under the same PCIe switch, the ACS ReqRedir and CmpltRedir bits force their peer-to-peer traffic upstream to the CPU root complex instead of routing it directly through the switch. If the IOMMU is disabled, or the root complex does not reflect the transaction, the NIC can no longer reach GPU memory and transfers fail with IBV_WC_REM_OP_ERR, completion status 11.

The symptom is counterintuitive: the GPU physically closest to the NIC fails while GPUs on other PCIe switches keep working.

Read the GPU-to-NIC topology. Pairs reported as PIX, meaning they share a PCIe switch, are where ACS redirect bites:

nvidia-smi topo -m

Check whether the IOMMU is off. With the IOMMU off, same-switch ACS redirect has nowhere to translate to:

cat /proc/cmdline | tr ' ' '\n' | grep -i iommu

Inspect ACS on a bridge in the data path. ReqRedir+ CmpltRedir+ with DirectTrans- is the problem state:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

Clear ACS redirect on every bridge in the GPU and NIC data path. The following systemd one-shot walks the PCIe path of each NVIDIA GPU and Mellanox NIC and clears ACS on the bridges it finds, so the fix survives a reboot and covers every GPU.

Write the script:

sudo tee /usr/local/sbin/disable-pcie-acs.sh >/dev/null <<'EOF'
#!/bin/bash
# Clear PCIe ACS Redirect on every bridge in the NVIDIA-GPU / RDMA-NIC data path
# so GPUDirect RDMA peer-to-peer DMA works for all GPUs. Clearing ACSCtl (offset
# +6 in the ACS extended capability) drops ReqRedir/CmpltRedir.
set -u
endpoints=$(for d in $(lspci -Dn | awk '{print $1}'); do
    v=$(cat "/sys/bus/pci/devices/$d/vendor" 2>/dev/null)
    [ "$v" = "0x10de" ] || [ "$v" = "0x15b3" ] && echo "$d"   # NVIDIA / Mellanox
done)
declare -A bridges
for ep in $endpoints; do
    for bdf in $(readlink -f "/sys/bus/pci/devices/$ep" | grep -oE '0000:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]'); do
        [ "$bdf" = "$ep" ] && continue
        case "$(cat "/sys/bus/pci/devices/$bdf/class" 2>/dev/null)" in
            0x0604*) bridges[$bdf]=1 ;;   # PCI bridges only
        esac
    done
done
for b in "${!bridges[@]}"; do
    cur=$(setpci -s "$b" ECAP_ACS+6.w 2>/dev/null) || continue   # no ACS capability
    [ "$cur" != "0000" ] && setpci -s "$b" ECAP_ACS+6.w=0000 2>/dev/null \
        && logger -t disable-pcie-acs "ACS cleared on $b (was $cur)"
done
EOF
sudo chmod +x /usr/local/sbin/disable-pcie-acs.sh

Write the unit:

sudo tee /etc/systemd/system/disable-pcie-acs.service >/dev/null <<'EOF'
[Unit]
Description=Disable PCIe ACS Redirect on GPU/RDMA-NIC fabric for GPUDirect RDMA P2P
After=sysinit.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/disable-pcie-acs.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now disable-pcie-acs.service

systemctl is-enabled reporting enabled is not sufficient. After the next reboot, confirm the job actually ran:

sudo journalctl -u disable-pcie-acs.service -b

The output must show Starting and Finished lines for the current boot, not -- No entries --.

Check that systemd did not drop the job. systemd words the message two ways depending on which unit it names first, so match both:

sudo journalctl -b | grep -E 'disable-pcie-acs.*ordering cycle|ordering cycle.*disable-pcie-acs'

Any hit means the job was dropped and ACS is still on.

Ground truth is the bridge state itself. The bridges above the GPU and the NIC should report ReqRedir- and CmpltRedir-:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

This is the single most important fabric requirement for inter-node RDMA at scale.

The inter-node transport paces itself with a per-peer congestion window and does not collapse on a lossy fabric. But the all-to-all erasure-write pattern, where every node writes shards to every other node, causes incast: many senders converge on one receiver’s switch egress port, overrun its buffer, and packets drop. Dropped packets surface as IBV_WC_RETRY_EXC_ERR and throughput falls away. Only a lossless fabric, meaning PFC plus DCQCN, sustains line rate under incast.

Configure losslessness end to end, identically on every host and every switch in the path. Host-only configuration does nothing if the switch drops.

Three layers must all agree on the same priority and DSCP value. The examples below use switch priority 3 and DSCP 26, the NVIDIA convention. If any layer is on a different class, losslessness does nothing.

Configure every switch in the path. The commands below are for NVIDIA Spectrum switches running Cumulus Linux 5.x with NVUE, the common 400G top-of-rack for ConnectX adapters. Adapt them for other vendors; the host and validation steps that follow are vendor-independent.

ssh cumulus@<switch>

nv config save

nv set qos roce mode lossless

nv set qos congestion-control default-global traffic-class 3 min-threshold 4194304
nv set qos congestion-control default-global traffic-class 3 max-threshold 41943040

nv config apply
nv config save

nv set qos roce mode lossless applies the validated profile: PFC on switch priority 3, ECN and RED on the lossless pool, DSCP 26 mapped to switch priority 3, and a dedicated lossless buffer.

The two congestion-control commands raise the ECN and RED marking thresholds to 4 MB and 40 MB. Raising them matters. The defaults of roughly 146 KB and 1.43 MB are tuned for low-speed links and over-throttle DCQCN at 400G; MinIO measured them capping a clean fabric at about 104 Gb/s per node. Raising the thresholds by roughly 10x lets the pipe fill before DCQCN marks, with PFC remaining the no-drop backstop.

Verify:

nv show qos roce
nv show qos congestion-control default-global traffic-class 3

nv show qos roce should report the feature enabled, mode lossless, and PFC on priority 3. The congestion-control output should report the 4 MB minimum and 40 MB maximum.

To roll back, run nv unset qos roce && nv config apply, or apply the snapshot saved by the first nv config save.

Apply these on every node. Replace <dev> with the RoCE netdev, for example ens257f0np0:

mlnx_qos -i <dev> --trust dscp
mlnx_qos -i <dev> --pfc 0,0,0,1,0,0,0,0
mlnx_qos -i <dev> --dscp2prio set,26,3
mlnx_qos -i <dev> --prio2buffer 0,0,0,1,0,0,0,0

The four settings mean, in order: trust DSCP so the marking survives L3 hops on routed RoCE v2; enable PFC on priority 3 only; map DSCP 26 to priority 3; and map priority 3 to the lossless buffer.

Confirm the result reports PFC on priority 3 and DSCP trust:

mlnx_qos -i <dev>

DCQCN is usually on by default. Confirm it on priority 3, where both files should read 1:

cat /sys/class/net/<dev>/ecn/roce_rp/enable/3
cat /sys/class/net/<dev>/ecn/roce_np/enable/3

mlnx_qos settings do not survive a reboot. Install a systemd one-shot that reapplies them at boot.

Write the script:

sudo tee /usr/local/sbin/roce-lossless.sh >/dev/null <<'EOF'
#!/bin/bash
# Apply the lossless RoCE profile to every RDMA adapter on this host.
# Exit non-zero if any adapter fails, so systemd reports the unit as failed
# rather than leaving a rail silently unconfigured.
rc=0
for ibdev in /sys/class/infiniband/*; do
    [ -d "$ibdev/device/net" ] || continue
    for dev in $(ls "$ibdev/device/net" 2>/dev/null); do
        for args in "--trust dscp" \
                    "--pfc 0,0,0,1,0,0,0,0" \
                    "--dscp2prio set,26,3" \
                    "--prio2buffer 0,0,0,1,0,0,0,0"; do
            if ! mlnx_qos -i "$dev" $args; then
                logger -t roce-lossless "FAILED on ${ibdev##*/} ($dev): mlnx_qos $args"
                rc=1
            fi
        done
        for rp in roce_np roce_rp; do
            f="/sys/class/net/$dev/ecn/$rp/enable/3"
            if ! echo 1 > "$f" 2>/dev/null; then
                logger -t roce-lossless "FAILED on ${ibdev##*/} ($dev): cannot write $f"
                rc=1
            fi
        done
    done
done
exit $rc
EOF
sudo chmod +x /usr/local/sbin/roce-lossless.sh

Write the unit:

sudo tee /etc/systemd/system/roce-lossless.service >/dev/null <<'EOF'
[Unit]
Description=RoCE lossless host QoS (PFC prio3 + DSCP26 + ECN)
After=network-online.target openibd.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/roce-lossless.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now roce-lossless.service

The script walks every adapter under /sys/class/infiniband, so a multi-NIC host has all of its RoCE rails configured. It exits non-zero if any adapter fails, so systemctl status roce-lossless reports the failure instead of leaving one rail silently lossy. Failures name both the RDMA device and its network interface in the journal, which matters on a multi-NIC host where the two differ:

journalctl -t roce-lossless

After the service runs, confirm each adapter took the settings:

for dev in /sys/class/infiniband/*/device/net/*; do mlnx_qos -i "$(basename "$dev")"; done

You do not hardcode a DSCP value in AIStor.

With the default setting MINIO_RDMA_INTERNODE_TRAFFIC_CLASS=0, meaning AUTO, the inter-node RDMA library reads the host’s Data Center Bridging configuration at NIC open through the dcbnl netlink interface. It reads the IEEE 802.1Qaz PFC enable bitmask and the Application Priority table, which holds the DSCP-to-priority map under selector APP_SEL_DSCP. From those it finds the PFC-enabled lossless priority and the DSCP that maps to it, then sets the GRH traffic class, which is the DSCP shifted left by 2, and the service level to match.

If no PFC class is configured, it falls back to best-effort. This is the same data that dcb app show and mlnx_qos report, queried directly from the kernel.

Set MINIO_RDMA_INTERNODE_TRAFFIC_CLASS only to override the detected value.

Once the switch and host steps above are done, inter-node RDMA uses the lossless class on its own.

Read the counters under load. The polarity is counterintuitive: rising pause counters are the good signal, because they mean the fabric is pausing instead of dropping.

Pause counters on priority 3 should be non-zero and rising under incast:

ethtool -S <dev> | grep -iE 'prio3_pause'

Drop counters must stay flat. These are the source of IBV_WC_RETRY_EXC_ERR:

ethtool -S <dev> | grep -iE 'discard|drop|out_of_buffer|rx_discards_phy'
grep -r . /sys/class/infiniband/<rdma_dev>/ports/1/hw_counters/ | \
    grep -iE 'out_of_buffer|packet_seq_err|local_ack_timeout_err'

If packet_seq_err or local_ack_timeout_err climb under load, the fabric is dropping and is not yet lossless.

Data should also egress on priority 3. Under load, tx_prio3_bytes rises while tx_prio0_bytes stays flat:

ethtool -S <dev> | grep -E 'tx_prio[03]_bytes'

With the fabric configured, install the RDMA build and point the service at it. See Deploy the RDMA server.