InfiniBand setup runbook

MemKV moves KV blocks over Remote Direct Memory Access (RDMA) on two fabrics: RoCEv2 on Ethernet, and native InfiniBand. You do not choose between them. MemKV reads the port's link layer at startup and addresses peers the way that fabric requires.

The difference is how a peer is named. InfiniBand routes by LID (Local Identifier), a 16-bit address the subnet manager assigns to every port in the subnet. RoCEv2 has no LIDs and instead carries the destination GID (Global Identifier) inside a global routing header. Using the wrong one builds an address the fabric cannot resolve, so nothing arrives and every transfer ends in a timeout.

This page covers what an InfiniBand deployment needs beyond the defaults. For Ethernet fabrics, read the RoCEv2 setup runbook instead.

What InfiniBand requires

Three things must be true before MemKV can move data. All three are properties of the fabric, not of MemKV.

MemKV tolerates the first two being late. A node that starts before the subnet manager reaches its port logs a warning, keeps running, and re-reads the LID on first use. This is deliberate: subnet manager assignment is a boot-order race that resolves itself within seconds, and failing at startup would disable the device for the life of the process.

What MemKV does not do is address a peer it cannot reach. Until the LID is known, transfers fail with a named error rather than a silent timeout.

MemKV also watches the device's asynchronous event queue, so a LID the subnet manager reassigns is picked up as well: on IBV_EVENT_LID_CHANGE — and on an SM change or a re-registration request, either of which can carry a new LID — it drops the cached value and re-reads it on next use. No restart is needed for new connections to use the right LID.

Which knobs apply

Several RDMA settings exist for one fabric only. MemKV applies each where it means something and reports the ones it ignores, but a configuration copied from a RoCEv2 cluster still needs review.

service_level is the one to watch when migrating. On RoCEv2 it names the lossless PFC priority, so a working Ethernet cluster often pins it to 3. On InfiniBand the same number selects a virtual lane through the subnet manager's SL-to-VL map, which is unrelated. Carry the value over only if you mean it in InfiniBand terms; otherwise leave it at 0.

InfiniBand provides lossless delivery through credit-based flow control in the fabric itself, so it needs no equivalent of the PFC and ECN configuration the RoCEv2 runbook describes. MemKV skips its DCB detection on InfiniBand ports.

Partitions

InfiniBand fabrics isolate tenants with partitions. The subnet manager gives each port a P_Key table, and every queue pair joins the partition named by one index in that table. Two ports in different partitions cannot exchange data.

MemKV uses index 0, the default partition, which is the right answer on an unpartitioned fabric and on RoCEv2. On a partitioned fabric, set the index your node's partition occupies:

rdma:
  pkey_index: 1

Clients take the same setting through MEMKV_RDMA_PKEY_INDEX or the pkey_index field in MEMKV_CONFIG.

Read the port's table to find the right index:

# each file is one table entry; 0x0000 means this node is not a member
cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/0

The top bit of the value is membership: set means full, clear means limited. Two limited members cannot reach each other.

memkv net show reports the same fact without reading sysfs by hand:

pkey_index        : 1   (full membership)

MemKV refuses to bring RDMA up on an empty entry, naming the index, rather than starting queue pairs on a partition where packets are discarded at the far end with no error on the sender. The server keeps running and serves over its non-RDMA fallback pool; clients drop that rail. Both log the reason at startup, so check the logs rather than assuming a healthy-looking server is on RDMA.

Same-subnet only

MemKV addresses InfiniBand peers by LID, which is meaningful only inside one subnet. Every MemKV server and client must sit in the same InfiniBand subnet.

A cross-subnet path needs a global routing header aimed at the local router, which MemKV does not build. It detects the mistake by comparing the two subnet prefixes and refuses the connection, naming both in the log of the side that built the address vector — so this fails at connect time with a clear cause instead of as an unexplained stall.

MTU

InfiniBand negotiates the same path MTU values as RoCEv2 — 256, 512, 1024, 2048, or 4096 bytes — and MemKV defaults to 4096. Unlike Ethernet, no jumbo-frame configuration is involved: the value comes from the port's negotiated active MTU.

MemKV refuses to start when the configured mtu exceeds what the port negotiated, naming both values. Check the port's active MTU with ibstat or memkv net show, and set the same mtu on servers and clients.

Verification

Confirm the subnet manager is running

sminfo
# reports the master SM's LID and priority; an error here means no SM,
# so no port in the subnet has a LID

Confirm the port is ready

memkv net show --device mlx5_0

The first two lines are the ones that matter:

device            : mlx5_0  port 1
link layer        : InfiniBand   (LID 33055)
transport         : DC + RC (mlx5)
pkey_index        : 0   (full membership)
service_level     : 0   (virtual lane, via the SM's SL-to-VL map)
path MTU          : 4096

link layer must read InfiniBand with a LID. If it reports no LID, the subnet manager has not configured the port — check sminfo first. If it reports Ethernet / RoCE on hardware you expect to be InfiniBand, the port is running in Ethernet mode; change the link protocol on the HCA.

ibstat gives the same facts from the fabric's side, including port state:

ibstat mlx5_0
# State: Active ; Physical state: LinkUp ; Base lid: 33055

Prove the data plane

memkv net verify --server node2:7788 --device mlx5_0 --auth-key <hex>

This connects to a running MemKV server, reports the transport actually negotiated, and writes and reads back a block to prove data moves. A VERIFIED line means the addressing, partition, and MTU all agree end to end.

Confirm raw bandwidth between two nodes

# server node
ib_write_bw -d mlx5_0 -a

# client node
ib_write_bw -d mlx5_0 -a node1

Expect near line rate with no retries. Unlike RoCEv2 there is no -x GID index to select, because InfiniBand addresses by LID.

Troubleshooting