Transport & Auth

MemKV uses the same authenticated wire format over two transports: RDMA and TCP. The full server on Linux always listens on both, and each client chooses which to use.

RDMA (DC + RC)

Data operations use DC (Dynamically Connected) transport on Mellanox mlx5 NICs that support DC. In classic RC, every pair of peers needs its own queue pair (QP), so the number of QPs grows as O(N²) with the cluster. DC replaces this with a shared pool of DCI (DC Initiator) QPs that can reach any DCT (DC Target) endpoint, so the number of QPs grows as O(N).

On hardware without DC support, MemKV falls back to RC (Reliable Connection) QPs automatically. The DCI pool has a fixed size by default (see rdma.num_dcis in Configuration). Under heavy fan-in, different workers can take turns on the same DCI.

Wire carriers

The server listens on TCP at the configured network.address (default 9900).

An RDMA client starts on that TCP connection: it sends Connect=0x06 over TCP, because the RC QP does not exist yet. Once Connect moves the RC QP to the ready-to-send (RTS) state, the client sends its control messages (Allocate, Lookup, Commit, Delete, Read, Write, BatchRead, BatchWrite, Exists) as RC SEND/RECV on that connection's QP. Bulk payloads move by RDMA WRITE and RDMA READ through the DCI/DCT pool, or over RC when DC is not available.

A client that does not use RDMA sends everything over the TCP connection. The inline-bulk codes (TcpPut=0x20, TcpGet=0x21, TcpDelete=0x22) and Exists carry block payloads and queries inside the signed, length-prefixed frame. A client with no RDMA NIC, or one set to transport: tcp, never sets up a QP.

The client chooses its transport with MEMKV_TRANSPORT (or transport: in MEMKV_CONFIG):

• auto tries RDMA first. On the first RDMA failure, at startup or during a transfer, it switches to TCP for the life of the engine and logs a WARN message. This is the default on Linux.
• rdma is strict. The client fails at startup if rdma_devices is empty or the build is not for Linux, and RDMA failures during a transfer are returned as errors. Use it in production when RDMA must carry the data.
• tcp skips RDMA. The client sends TCP traffic to the configured server address. This is the default on macOS.

The transport is the client's choice. The server has no setting to turn off either listener.

Topology vs storage mode

These are two independent choices:

• Topology is either distributed (several servers, each with its own NVMe drives; the shape behind the 97.4 GB/s results in Benchmarks) or co-located (one MemKV server on the inference host, common on GPU servers with NVMe next to the NIC).
• Storage mode is either direct or file. direct uses raw block devices and is the high-performance mode in either topology. file stores data in regular files through mmap, for developers, kind or CI clusters, Macs, and hosts with no raw drives to dedicate. In file mode the host filesystem limits the bandwidth.

macOS always runs in file mode over TCP, because the macOS build has no RDMA, JBOF, or hugepage support. You can run the same shape on Linux with storage.mode: file and transport: tcp, but a Linux host with raw drives should use direct for full performance.

Authentication

Every signed wire message ends with a 40-byte trailer: an 8-byte timestamp followed by a 32-byte HMAC-SHA256 over (header || ts_ns). Both ends use the same shared key: network.auth_key on the server, auth_key: on the client, or MEMKV_AUTH_KEY on either. The allowed clock drift is ±60 s. MemKV drops a message without a reply if its timestamp is outside that window or its MAC is wrong. There is no unauthenticated mode: the engine refuses to start without a key.

Encryption

MemKV handles authentication and encryption separately, at different layers.

On the wire. The HMAC trailer covers only the control header and its timestamp. It proves that the sender holds the shared key and rejects replays outside the drift window. It does not cover the payload, so it does not keep the data secret or show that someone changed it. TCP frames carry a CRC32, which detects corruption but not a deliberate rewrite.

RDMA usually runs over a dedicated RoCE fabric, on a storage network that is physically isolated. On the TCP path the KV data also travels unencrypted, and the TCP data plane does not support TLS. Use TCP only on trusted networks, and keep it off links you do not control. The tls_cert and tls_key settings apply only to the admin HTTP endpoint.

On the drive. Encryption at rest works with either transport. Set crypto.enabled: true and put a 32-byte AES-256 master key in crypto.master_key_hex. The storage layer then encrypts every sector with AES-256-GCM on write and decrypts it on read.

MemKV can also run the AES-256-GCM work on an NVIDIA BlueField-3 DPU through DOCA, which keeps the cipher off the host CPUs. This needs a MemKV build with DOCA support, with crypto.doca_device set to the device's PCI address. Without that build, without crypto.doca_device, or without an AES-GCM-capable device, MemKV runs the same encryption in software.

Context block offload flow

1. Dynamo/KVBM picks context blocks to offload.
2. NIXL calls the MemKV plugin with the client buffer address and rkey.
3. The plugin sends a write request to the server through RDMA messaging (RC QP).
4. The server takes a DCI from the pool and uses RDMA READ to copy the data from client memory (DC transport).
5. The server writes the data to NVMe through io_uring with O_DIRECT.
6. On a lookup hit, the server reads the data from NVMe and uses RDMA WRITE through a DCI to send it back to the client.