InfiniBand setup runbook

MemKV moves KV blocks over Remote Direct Memory Access (RDMA) on two fabrics: RoCEv2 on Ethernet, and native InfiniBand. You do not choose between them. At startup, MemKV reads the port's link layer and addresses peers the way that fabric requires.

On InfiniBand, MemKV addresses each peer by its LID (Local Identifier), a 16-bit address that the subnet manager assigns to every port in the subnet.

This page covers what an InfiniBand deployment needs beyond the defaults. For Ethernet fabrics, read the RoCEv2 setup runbook instead.

What InfiniBand requires

The fabric must provide three things before MemKV can move data:

If MemKV starts before the subnet manager assigns a LID, it logs a warning, keeps running, and reads the LID again when it first needs it. You do not need to restart MemKV.

If the subnet manager assigns a new LID to a running port, new connections use the new LID without a restart. Existing sessions fail and reconnect, so expect a short burst of transfer errors and reconnections when the subnet manager renumbers a live fabric.

Settings that differ from RoCEv2

Some RDMA settings apply to only one fabric. MemKV ignores the ones that do not apply, but review any configuration you copy from a RoCEv2 cluster.

Check service_level when you move a configuration. A RoCEv2 cluster often sets it to 3, its lossless priority. On InfiniBand, the same number selects a virtual lane through the subnet manager's SL-to-VL map, which has a different meaning. Unless you have chosen a virtual lane, set it to 0.

InfiniBand is lossless by design, so it needs none of the PFC and ECN setup in the RoCEv2 runbook.

Partitions

InfiniBand fabrics isolate tenants with partitions. The subnet manager gives each port a P_Key table, and every queue pair joins the partition at one index in that table. Ports in different partitions cannot exchange data.

MemKV uses index 0, the default partition. This is correct on a fabric without partitions. On a partitioned fabric, set the index of your node's partition:

rdma:
  pkey_index: 1

Clients take the same setting from MEMKV_RDMA_PKEY_INDEX or the pkey_index field in MEMKV_CONFIG.

To find the right index, read the port's table:

# each file is one table entry; 0x0000 means this node is not a member
cat /sys/class/infiniband/mlx5_0/ports/1/pkeys/0

The top bit of the value shows the membership type: set means full, clear means limited. Two limited members cannot reach each other. memkv net show reports the same information:

pkey_index        : 1   (full membership)

If the entry at pkey_index is empty, MemKV does not start RDMA and logs the index. The server keeps running without RDMA, and clients stop using that NIC. Check the startup log to confirm that RDMA is running.

Same subnet only

Every MemKV server and client must be in the same InfiniBand subnet. A LID is valid only inside one subnet, and MemKV does not route across subnets. If a client and server are in different subnets, the connection fails and the log names both subnet prefixes.

MTU

InfiniBand uses the same path MTU values as RoCEv2: 256, 512, 1024, 2048, or 4096 bytes. MemKV uses 4096 by default. You do not need jumbo frames: the value comes from the port's active MTU.

MemKV refuses to start when the configured mtu is larger than the port's active MTU, and the error names both values. Check the port's active MTU with ibstat or memkv net show, and set the same mtu on servers and clients.

Verification

Confirm the subnet manager is running

sminfo
# reports the master SM's LID and priority; an error here means no SM,
# so no port in the subnet has a LID

Confirm the port is ready

memkv net show --device mlx5_0

Check the first two lines:

device            : mlx5_0  port 1
link layer        : InfiniBand   (LID 33055)
transport         : DC + RC (mlx5)
pkey_index        : 0   (full membership)
service_level     : 0   (virtual lane, via the SM's SL-to-VL map)
path MTU          : 4096

link layer must show InfiniBand with a LID.

• If there is no LID, the subnet manager has not configured the port. Run sminfo.
• If it shows Ethernet / RoCE on InfiniBand hardware, the port is in Ethernet mode. Change the link protocol on the HCA.

ibstat shows the same information from the fabric, including the port state:

ibstat mlx5_0
# State: Active ; Physical state: LinkUp ; Base lid: 33055

Confirm data moves over RDMA

memkv net verify --server node2:7788 --device mlx5_0 --auth-key <hex>

This command connects to a running MemKV server, reports the transport in use, and writes and reads back a block. A VERIFIED line means the addressing, partition, and MTU are correct end to end.

Confirm raw bandwidth between two nodes

# server node
ib_write_bw -d mlx5_0 -a

# client node
ib_write_bw -d mlx5_0 -a node1

Expect close to line rate with no retries. InfiniBand needs no -x GID index.

Troubleshooting