Configure the RDMA fabric

Prepare the hardware, the switches, and the host operating system before you install the RDMA build of AIStor Server. An AIStor node with an unconfigured fabric starts normally and serves every request over TCP, so a missed step here shows up as “RDMA does nothing” rather than as an error.

Work through the sections in order. Each section names which of the two RDMA paths it applies to.

Both RDMA paths need the following:

On a RoCE fabric, inter-node RDMA additionally requires that the fabric be lossless end to end. See Make the RoCE fabric lossless.

On an InfiniBand fabric, inter-node RDMA addresses a peer by its local identifier (LID), which each node sends to the others. Run the same MinIO AIStor release on every node. During an upgrade, a node still on the earlier release sends no LID, and inter-node transfers to it behave as they did on that release. A RoCE fabric is unaffected, because the address travels in the packet header and the LID goes unused.

A 5.4 kernel meets the minimum, but the RDMA and GPUDirect paths have improved steadily since. Prefer 6.15 or later, which on Ubuntu 24.04 LTS means installing the hardware enablement (HWE) kernel rather than the release kernel.

Run this before anything else. If ibv_devinfo reports no device, or the port state is not PORT_ACTIVE, fix the fabric before you continue.

ibv_devinfo

The output lists each adapter and its port state:

hca_id: mlx5_0
  transport:      InfiniBand (0)
  fw_ver:         20.31.1014
  phys_port_cnt:  1
    port:   1
      state:        PORT_ACTIVE (4)
      max_mtu:      4096 (5)
      active_mtu:   4096 (5)

Check the link state and the fabric view as well:

ibstat
ibstatus

Load the RDMA modules on every node:

sudo modprobe ib_core
sudo modprobe ib_uverbs
sudo modprobe rdma_cm
sudo modprobe mlx5_core

Confirm they loaded:

lsmod | grep -E "ib_|rdma_|mlx"

To load them at every boot, create /etc/modules-load.d/rdma.conf:

ib_core
ib_uverbs
rdma_cm
mlx5_core

GPU-Direct RDMA moves the object payload directly between GPU memory and the RDMA NIC over PCIe peer-to-peer DMA. Many servers block this silently through PCIe Access Control Services (ACS) redirect.

When a GPU and the NIC sit under the same PCIe switch, the ACS ReqRedir and CmpltRedir bits force their peer-to-peer traffic upstream to the CPU root complex instead of routing it directly through the switch. If the IOMMU is disabled, or the root complex does not reflect the transaction, the NIC can no longer reach GPU memory and transfers fail with IBV_WC_REM_OP_ERR, completion status 11.

The symptom is counterintuitive: the GPU physically closest to the NIC fails while GPUs on other PCIe switches keep working.

Read the GPU-to-NIC topology. Pairs reported as PIX, meaning they share a PCIe switch, are where ACS redirect bites:

nvidia-smi topo -m

Check whether the IOMMU is off. With the IOMMU off, same-switch ACS redirect has nowhere to translate to:

cat /proc/cmdline | tr ' ' '\n' | grep -i iommu

Inspect ACS on a bridge in the data path. ReqRedir+ CmpltRedir+ with DirectTrans- is the problem state:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

Clear ACS redirect on every bridge in the GPU and NIC data path. The following systemd one-shot walks the PCIe path of each NVIDIA GPU and Mellanox NIC and clears ACS on the bridges it finds, so the fix survives a reboot and covers every GPU.

Write the script:

sudo tee /usr/local/sbin/disable-pcie-acs.sh >/dev/null <<'EOF'
#!/bin/bash
# Clear PCIe ACS Redirect on every bridge in the NVIDIA-GPU / RDMA-NIC data path
# so GPUDirect RDMA peer-to-peer DMA works for all GPUs. Clearing ACSCtl (offset
# +6 in the ACS extended capability) drops ReqRedir/CmpltRedir.
set -u
endpoints=$(for d in $(lspci -Dn | awk '{print $1}'); do
    v=$(cat "/sys/bus/pci/devices/$d/vendor" 2>/dev/null)
    [ "$v" = "0x10de" ] || [ "$v" = "0x15b3" ] && echo "$d"   # NVIDIA / Mellanox
done)
declare -A bridges
for ep in $endpoints; do
    for bdf in $(readlink -f "/sys/bus/pci/devices/$ep" | grep -oE '0000:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9a-f]'); do
        [ "$bdf" = "$ep" ] && continue
        case "$(cat "/sys/bus/pci/devices/$bdf/class" 2>/dev/null)" in
            0x0604*) bridges[$bdf]=1 ;;   # PCI bridges only
        esac
    done
done
for b in "${!bridges[@]}"; do
    cur=$(setpci -s "$b" ECAP_ACS+6.w 2>/dev/null) || continue   # no ACS capability
    [ "$cur" != "0000" ] && setpci -s "$b" ECAP_ACS+6.w=0000 2>/dev/null \
        && logger -t disable-pcie-acs "ACS cleared on $b (was $cur)"
done
EOF
sudo chmod +x /usr/local/sbin/disable-pcie-acs.sh

Write the unit:

sudo tee /etc/systemd/system/disable-pcie-acs.service >/dev/null <<'EOF'
[Unit]
Description=Disable PCIe ACS Redirect on GPU/RDMA-NIC fabric for GPUDirect RDMA P2P
After=sysinit.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/disable-pcie-acs.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now disable-pcie-acs.service

systemctl is-enabled reporting enabled is not sufficient. After the next reboot, confirm the job actually ran:

sudo journalctl -u disable-pcie-acs.service -b

The output must show Starting and Finished lines for the current boot, not -- No entries --.

Check that systemd did not drop the job. systemd words the message two ways depending on which unit it names first, so match both:

sudo journalctl -b | grep -E 'disable-pcie-acs.*ordering cycle|ordering cycle.*disable-pcie-acs'

Any hit means the job was dropped and ACS is still on.

Ground truth is the bridge state itself. The bridges above the GPU and the NIC should report ReqRedir- and CmpltRedir-:

sudo lspci -vvv -s <bridge-bdf> | grep ACSCtl

This is the single most important fabric requirement for inter-node RDMA at scale.

The inter-node transport paces itself with a per-peer congestion window and does not collapse on a lossy fabric. But the all-to-all erasure-write pattern, where every node writes shards to every other node, causes incast: many senders converge on one receiver’s switch egress port, overrun its buffer, and packets drop. Dropped packets surface as IBV_WC_RETRY_EXC_ERR and throughput falls away. Only a lossless fabric, meaning PFC plus DCQCN, sustains line rate under incast.

Configure losslessness end to end, identically on every host and every switch in the path. Host-only configuration does nothing if the switch drops.

Three layers must all agree on the same priority and DSCP value. The examples below use switch priority 3 and DSCP 26, the NVIDIA convention. If any layer is on a different class, losslessness does nothing.

Configure every switch in the path. The commands below are for NVIDIA Spectrum switches running Cumulus Linux 5.x with NVUE, the common 400G top-of-rack for ConnectX adapters. Adapt them for other vendors; the host and validation steps that follow are vendor-independent.

ssh cumulus@<switch>

nv config save

nv set qos roce mode lossless

nv set qos congestion-control default-global traffic-class 3 min-threshold 4194304
nv set qos congestion-control default-global traffic-class 3 max-threshold 41943040

nv config apply
nv config save

nv set qos roce mode lossless applies the validated profile: PFC on switch priority 3, ECN and RED on the lossless pool, DSCP 26 mapped to switch priority 3, and a dedicated lossless buffer.

The two congestion-control commands raise the ECN and RED marking thresholds to 4 MB and 40 MB. Raising them matters. The defaults of roughly 146 KB and 1.43 MB are tuned for low-speed links. At 400G they over-throttle DCQCN, which marks long before the pipe is full and holds throughput well below what the fabric can carry. Raising the thresholds by roughly 28 times lets the pipe fill before DCQCN marks, with PFC remaining the no-drop backstop.

Verify:

nv show qos roce
nv show qos congestion-control default-global traffic-class 3

nv show qos roce should report the feature enabled, mode lossless, and PFC on priority 3. The congestion-control output should report the 4 MB minimum and 40 MB maximum.

To roll back, run nv unset qos roce && nv config apply, or apply the snapshot saved by the first nv config save.

Apply these on every node. Replace <dev> with the RoCE netdev, for example ens257f0np0:

mlnx_qos -i <dev> --trust dscp
mlnx_qos -i <dev> --pfc 0,0,0,1,0,0,0,0
mlnx_qos -i <dev> --dscp2prio set,26,3
mlnx_qos -i <dev> --prio2buffer 0,0,0,1,0,0,0,0

The four settings mean, in order: trust DSCP so the marking survives L3 hops on routed RoCE v2; enable PFC on priority 3 only; map DSCP 26 to priority 3; and map priority 3 to the lossless buffer.

Confirm the result reports PFC on priority 3 and DSCP trust:

mlnx_qos -i <dev>

DCQCN is usually on by default. Confirm it on priority 3, where both files should read 1:

cat /sys/class/net/<dev>/ecn/roce_rp/enable/3
cat /sys/class/net/<dev>/ecn/roce_np/enable/3

mlnx_qos settings do not survive a reboot. Install a systemd one-shot that reapplies them at boot.

Write the script:

sudo tee /usr/local/sbin/roce-lossless.sh >/dev/null <<'EOF'
#!/bin/bash
# Apply the lossless RoCE profile to every RDMA adapter on this host.
# Exit non-zero if any adapter fails, so systemd reports the unit as failed
# rather than leaving a rail silently unconfigured.
rc=0
for ibdev in /sys/class/infiniband/*; do
    [ -d "$ibdev/device/net" ] || continue
    for dev in $(ls "$ibdev/device/net" 2>/dev/null); do
        for args in "--trust dscp" \
                    "--pfc 0,0,0,1,0,0,0,0" \
                    "--dscp2prio set,26,3" \
                    "--prio2buffer 0,0,0,1,0,0,0,0"; do
            if ! mlnx_qos -i "$dev" $args; then
                logger -t roce-lossless "FAILED on ${ibdev##*/} ($dev): mlnx_qos $args"
                rc=1
            fi
        done
        for rp in roce_np roce_rp; do
            f="/sys/class/net/$dev/ecn/$rp/enable/3"
            if ! echo 1 > "$f" 2>/dev/null; then
                logger -t roce-lossless "FAILED on ${ibdev##*/} ($dev): cannot write $f"
                rc=1
            fi
        done
    done
done
exit $rc
EOF
sudo chmod +x /usr/local/sbin/roce-lossless.sh

Write the unit:

sudo tee /etc/systemd/system/roce-lossless.service >/dev/null <<'EOF'
[Unit]
Description=RoCE lossless host QoS (PFC prio3 + DSCP26 + ECN)
After=network-online.target openibd.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/roce-lossless.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now roce-lossless.service

The script walks every adapter under /sys/class/infiniband, so a multi-NIC host has all of its RoCE rails configured. It exits non-zero if any adapter fails, so systemctl status roce-lossless reports the failure instead of leaving one rail silently lossy. Failures name both the RDMA device and its network interface in the journal, which matters on a multi-NIC host where the two differ:

journalctl -t roce-lossless

After the service runs, confirm each adapter took the settings:

for dev in /sys/class/infiniband/*/device/net/*; do mlnx_qos -i "$(basename "$dev")"; done

You do not hardcode a DSCP value in AIStor.

With the default setting MINIO_RDMA_INTERNODE_TRAFFIC_CLASS=0, meaning AUTO, the inter-node RDMA library reads the host’s Data Center Bridging configuration at NIC open through the dcbnl netlink interface. It reads the IEEE 802.1Qaz PFC enable bitmask and the Application Priority table, which holds the DSCP-to-priority map under selector APP_SEL_DSCP. From those it finds the PFC-enabled lossless priority and the DSCP that maps to it, then sets the GRH traffic class, which is the DSCP shifted left by 2, and the service level to match.

If no PFC class is configured, it falls back to best-effort. This is the same data that dcb app show and mlnx_qos report, queried directly from the kernel.

Set MINIO_RDMA_INTERNODE_TRAFFIC_CLASS only to override the detected value.

Once the switch and host steps above are done, inter-node RDMA uses the lossless class on its own.

Read the counters under load. The polarity is counterintuitive: rising pause counters are the good signal, because they mean the fabric is pausing instead of dropping.

Pause counters on priority 3 should be non-zero and rising under incast:

ethtool -S <dev> | grep -iE 'prio3_pause'

Drop counters must stay flat. These are the source of IBV_WC_RETRY_EXC_ERR:

ethtool -S <dev> | grep -iE 'discard|drop|out_of_buffer|rx_discards_phy'
grep -r . /sys/class/infiniband/<rdma_dev>/ports/1/hw_counters/ | \
    grep -iE 'out_of_buffer|packet_seq_err|local_ack_timeout_err'

If packet_seq_err or local_ack_timeout_err climb under load, the fabric is dropping and is not yet lossless.

Data should also egress on priority 3. Under load, tx_prio3_bytes rises while tx_prio0_bytes stays flat:

ethtool -S <dev> | grep -E 'tx_prio[03]_bytes'

Apply this to any host whose RoCE rails share one layer-2 broadcast domain. Rails in genuinely separate domains do not need it. Separate switches are not enough on their own, because a VLAN trunked between them is still one domain.

Apply it in advance rather than waiting to see a failure. Which rail breaks is a race: the requester keeps whichever ARP reply arrives last, so a rail can work for months and then stop after an unrelated reboot.

Scope the settings to the RoCE rails. The effective value is the larger of conf.all.X and conf.<interface>.X, so setting conf.all also changes the management NIC and every other interface on the host.

# one stanza per RoCE rail; substitute the real interface names
sudo tee /etc/sysctl.d/99-rdma-arp.conf >/dev/null <<'EOF'
# Answer ARP only for addresses on the receiving interface, and pick the
# source address for the target. Without this, a peer can cache one rail's
# address against another rail's MAC.
net.ipv4.conf.enp24s0np0.arp_ignore = 1
net.ipv4.conf.enp24s0np0.arp_announce = 2
net.ipv4.conf.enp196s0np0.arp_ignore = 1
net.ipv4.conf.enp196s0np0.arp_announce = 2
EOF

sudo sysctl --system

arp_ignore=1 answers only for addresses configured on the receiving interface. arp_announce=2 selects the best local address for the target. It is source selection, not a guarantee that the address belongs to the outgoing interface.

Setting conf.all instead is a valid shortcut, and is what most vendor guides show. Check first that it does not disturb a management NIC that relies on the default, such as a VIP or load-balancer setup.

The settings govern how addresses are resolved from now on. They do not rewrite entries already in the neighbor cache, so clear those too.

sudo ip neigh flush dev <rail>

flush leaves permanent and noarp entries alone, so a static mapping survives it and keeps pointing at the wrong rail. Look for them, and remove any by hand:

ip neigh show dev <rail> nud permanent
sudo ip neigh del <peer-rail-ipv4> dev <rail>

Flushing only discards entries. The kernel re-resolves an address the next time it has traffic for it, so send a packet over each rail to bring the entries back. Use -4: this procedure concerns the IPv4 neighbor cache, and a name that resolves to IPv6 populates a different table.

ping -4 -c1 -W2 -I <rail> <peer-rail-ipv4>

Each peer address must resolve to the MAC of the interface that owns it. Compare what this node learned:

ip neigh show dev <rail>

against the peer’s own interfaces:

# On the peer: which MAC each interface has, and which address it carries
ip -brief link show
ip -brief address show

A rail’s address resolving to another rail’s MAC is the fault described below. No other tool on the host reports it, so this comparison is the check that matters.

Linux resolves ARP per host, not per interface. With the default arp_ignore=0, a node answers an ARP request for any of its local addresses on whichever interface hears the request. When two rails share a broadcast domain, a request for rail 1’s address reaches the node on both rails, and either may answer. The requester can therefore cache rail 1’s address against rail 2’s MAC. Giving each rail its own subnet does not prevent this, because arp_ignore=0 answers without regard to the sender’s subnet.

TCP usually tolerates the wrong MAC. By default the kernel accepts a frame for any local address, whichever interface it arrived on, so ping, SSH, the internode gRPC channel, and the HTTP S3 path all keep working, and nothing in the logs suggests a problem. Host policy can change that: reverse-path filtering, for example, can drop traffic arriving on an unexpected interface. Healthy TCP therefore does not rule this fault out.

RoCE does not tolerate it. A queue pair is bound to one device and GID, so a frame delivered to the wrong rail carries a GID that does not live on the receiving port. The adapter drops it, nothing acknowledges it, and the sender exhausts its transport retries and fails with IBV_WC_RETRY_EXC_ERR.

Where the topology allows it, giving each rail its own VLAN or its own switch is the stronger fix, because it removes the ambiguity rather than suppressing it. This is the usual rail-optimized design for RoCE fabrics. Separate subnets on a shared broadcast domain is the combination that can fail, because the subnets give a false sense of separation that ARP does not honor.

With the fabric configured, install the RDMA build and point the service at it. See Deploy the RDMA server.