RoCEv2 setup runbook

MemKV moves KV blocks over Remote Direct Memory Access (RDMA). On Ethernet, MemKV uses RoCEv2 (RDMA over Converged Ethernet, version 2). RoCEv2 performs well only when the switch, the host NIC, and MemKV agree on three settings:

• A lossless priority: Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) on one traffic priority.
• A marking: the DSCP value and service level that put RDMA traffic on that priority.
• An MTU that every hop can carry.

If one of them is wrong, RDMA traffic either goes to the best-effort queue (slow and lossy) or is dropped, and the client falls back to TCP.

This page gives one setup path for NVIDIA ConnectX-5 or later (mlx5) NICs on Linux. Adapt the switch steps to your switch vendor.

The three things that must agree

This page uses the common RoCE marking, DSCP 26 on priority 3. MemKV auto-detection prefers this pair when it finds it. Use it unless your fabric already uses another lossless priority.

Why MTU is the common failure

RoCE uses a path MTU of 256, 512, 1024, 2048, or 4096 bytes. The NIC derives it from the Ethernet MTU of the link. A standard 1500-byte Ethernet link gives a RoCE path MTU of 1024. To get 4096, every hop needs jumbo frames: an Ethernet MTU of about 4200 or more. In practice, set 9000.

MemKV uses mtu: 4096 by default. MemKV refuses to start when the configured mtu is larger than the port's active MTU, and the error names both values. So either enable jumbo frames end to end, as the steps below do, or lower mtu to match the port.

A single-node test never puts a packet on the wire, so MTU and PFC errors only show up between two nodes. Always test across nodes.

Switch configuration

Make the RoCE priority lossless, enable ECN, and trust the DSCP marking from the hosts. On NVIDIA Spectrum switches with Cumulus Linux:

# Enable PFC on priority 3 (the RoCE priority)
nv set qos pfc default-global pfc-priority 3
nv set qos pfc default-global state enable

# Trust the incoming DSCP marking and map DSCP 26 -> traffic class 3
nv set qos mapping default-global trust dscp
nv set qos mapping default-global dscp 26 switch-priority 3

# Enable ECN (WRED) on the RoCE traffic class for congestion signaling
nv set qos congestion-control default-global congestion-mode ecn
nv set qos congestion-control default-global traffic-class 3

# Jumbo frames on every RoCE-facing port
nv set interface swp1-swp32 link mtu 9216
nv config apply

Host (NIC) configuration

Run these steps on each host, on the network interface of the RDMA NIC. To find the interface, run rdma link or ibdev2netdev. The examples assume that mlx5_0 maps to enp1s0f0.

mlnx_qos -i enp1s0f0 --trust dscp
mlnx_qos -i enp1s0f0 --dscp2prio set,26,3

# bitmask per priority 0..7 — enable PFC on priority 3 only
mlnx_qos -i enp1s0f0 --pfc 0,0,0,1,0,0,0,0

echo 1 > /sys/class/net/enp1s0f0/ecn/roce_np/enable/3
echo 1 > /sys/class/net/enp1s0f0/ecn/roce_rp/enable/3

ip link set enp1s0f0 mtu 9000

MemKV configuration

When the fabric is lossless and uses jumbo frames, keep the defaults. With a value of 0, MemKV reads the DSCP-to-priority map and PFC settings from the NIC and sets the matching DSCP and service level on outgoing traffic.

rdma:
  device: mlx5_0
  gid_index: 0 # 0 = auto-detect the routable RoCEv2 GID
  traffic_class: 0 # 0 = auto-detect the lossless DSCP (e.g. 104 for DSCP 26)
  service_level: 0 # 0 = take the priority paired with the DSCP (e.g. 3)
  mtu: 4096 # requires jumbo frames end-to-end (above)

Some virtualized or non-mlx5 NICs do not expose these settings to MemKV. On those NICs, set the values yourself:

• gid_index: a RoCEv2 entry in the GID table. Find it with show_gids (see Verification). On mlx5, index 0 is a RoCEv1 entry, which carries no DSCP and cannot reach the lossless queue.
• traffic_class: the IP ToS byte, which is DSCP << 2. For DSCP 26, it is 104.
• service_level: the lossless PFC priority, 3. On mlx5 the NIC queues traffic by service level, not by DSCP, so a wrong service level sends traffic to the best-effort queue even when the DSCP is right.

rdma:
  gid_index: 3 # a routable RoCEv2 GID entry (see verification below)
  traffic_class: 104 # DSCP 26 << 2
  service_level: 3 # matches the lossless PFC priority
  mtu: 4096

Verification

memkv net runs most of these checks with the values MemKV itself uses:

• memkv net show reports the GID index, DSCP and traffic class, service level, and path MTU.
• memkv net probe takes a snapshot of the PFC counters for each priority.
• memkv net verify --server <ip:port> --auth-key <hex> runs an RDMA round trip to a running server.

When a check fails, use the manual steps below to find the layer that is wrong.

ibv_devinfo -d mlx5_0 | grep -E 'active_mtu|state'
# active_mtu: 4096 (5)   <- jumbo working; 1024 (3) means MTU too low

show_gids mlx5_0
# look for RoCE v2 rows with a v4 (::ffff:a.b.c.d) IP — that index is what
# auto-detect picks; pin it as gid_index if you set it by hand

mlnx_qos -i enp1s0f0
# trust: dscp ; pfc enabled on prio 3 ; dscp 26 -> prio 3

ethtool -S enp1s0f0 | grep -E 'prio3|prio0' > before.txt
# run a MemKV read/write workload, then:
ethtool -S enp1s0f0 | grep -E 'prio3|prio0' > after.txt
diff before.txt after.txt
# tx_prio3_bytes must climb; tx_prio0_bytes should not. If prio0 moves and
# prio3 doesn't, the SL/DSCP isn't landing traffic on the lossless queue.

# server node
ib_write_bw -d mlx5_0 -x 3 --report_gbits
# client node
ib_write_bw -d mlx5_0 -x 3 --report_gbits <server-ip>
# -x 3 selects a RoCEv2 GID index; expect near line rate with no retries

Troubleshooting