Benchmarks

This page measures the wire: how fast bytes move between a client and MemKV. For what that speed gives an inference server under a production-shaped agentic workload (continuing-turn TTFT, tok/s/GPU, and the cost below the capacity cliff), see Inference performance.

Throughput by block size (2-server aggregate)

Test harness. 1 GPU node (2× ConnectX-7 400GbE) drove 2 storage servers (12 NVMe each) with the memkv bench binary, at 64 threads per server (128 total on the dual-server sweep). RDMA and TCP were swept separately. The TCP sweep used the inline-bulk path with 64 client connections (one per thread). Last verified on 2026-07-06.

RDMA peak: 97.4 GB/s read, 97.4 GB/s write, about 97% of 2× 400GbE line rate. TCP peak: 22.3 GB/s read, 16.3 GB/s write. RDMA runs near line rate from about 1 MB blocks up. TCP still reaches tens of GB/s by pipelining many requests over each connection, so it is a practical transport wherever RDMA cannot reach end to end: routed or multi-hop fabrics, cloud, and mixed NICs.

Linear scaling

Each server has 12 NVMe drives in the same PCIe domain as the NIC. The servers share no drives and do not coordinate, so adding servers scales throughput linearly.

Under live engine load

The sweeps above measure the store on its own. The August 2026 NVIDIA AIPerf agentic campaign measured it under a live inference engine: the same 198 windows on 8× H200 and 2 MemKV nodes (48 raw NVMe, dual 400 Gb rails) that feed Inference performance. The storage tier was never the bottleneck:

In the same campaign, NIXL nixlbench on KV-shaped blocks over dual-rail RoCE sustained 104 GB/s.

NIXL plugin (nixlbench)

These results come from NVIDIA's nixlbench (from nixl v1.2.0) driving the MemKV NIXL plugin against the same dual-server fleet as the native sweep, with DRAM destinations. Every run uses --recreate_xfer, which creates a new transfer on every iteration so no cached handle speeds it up. Block sizes go up to nixlbench's default maximum of 64 MiB.

Inside each post_xfer call, the plugin merges the descriptor list: descriptors for the same key at contiguous offsets become one wide BatchRead, and the server moves the data in 2 MiB bounce-buffer chunks.

These are separate sweeps from the native memkv bench results at the top of this page (97.4 GB/s peak) and from the 104 GB/s KV-shaped figure in Under live engine load.

Batched write throughput (GB/s)

nixlbench WRITE, batch_size=16, --recreate_xfer, 64 threads, swept to the default 64 MiB max block size:

The writes in a batch persist to the NVMe drives in parallel, so batched WRITE throughput grows with block size. At 64 MB a single rail reaches about 89% of its line rate.

Read throughput scales across rails the same way. The figures at the top of this page (48.7 GB/s for one server, 97.4 GB/s for two) are RDMA reads. The plugin adds one feature for reads: it merges reads of a shared range.

Read coalescing: fetch once, scatter locally

The plugin merges (coalesces) a list of read descriptors before it sends anything over the network. When several descriptors in one post_xfer point to the same KV block with overlapping or contiguous ranges, the plugin:

1. Picks one destination that spans the whole range as the cover.
2. Issues a single RDMA read into the cover.
3. Fills every other destination with a local copy from the cover: std::memcpy for host (DRAM) buffers, or cudaMemcpy for GPU (VRAM) buffers through the CUDA runtime, which the plugin loads at run time.

The shared block crosses the NIC once and reaches N buffers at local memory speed (DRAM or HBM), not network speed.

This is what makes prefix sharing pay off: when many concurrent sequences read the same prompt prefix, the prefix crosses the fabric once and is copied to each reader in memory. If the destinations are GPU memory and no CUDA copy path is available, or no single destination covers the range, the plugin sends a separate network read for each destination, so the result stays correct.

Writes are never merged this way, because writing one object range from two source buffers has no defined result. Each write persists on its own, so write throughput shows the real network and NVMe bandwidth (see the table above).

Why read throughput can exceed line rate

When you benchmark the plugin with nixlbench, it can report read throughput well above NIC line rate: for example, about 150 GB/s on a single 400GbE rail at batch_size=16, roughly 3× the rail's ~50 GB/s limit. This is expected. It is read coalescing showing through the benchmark's byte counting, not a measurement error:

• nixlbench credits every descriptor in a batch with a full block transfer. It counts the logical bytes delivered to the caller.
• The plugin keys each descriptor by its NIXL dev_id, and the nixlbench synthetic workload uses very few distinct keys (one when num_initiator_dev=1). So a batch's descriptors all point to the same KV block, which the plugin fetches once and copies to every destination locally.
• Reported throughput is logical bytes divided by time. The shared bytes cross the NIC once but are counted N times, so the reported rate is above the network limit.

The bytes are real: every destination receives correct data (--check_consistency=1 passes), and the gain is real for reads that share a prefix. But it is an effective rate, limited by local memory bandwidth for the shared part, not by network bandwidth. A workload where every key is different gets no coalescing, and its reads follow the network-limited rates below.

Per-op read wire rate (batch_size=1)

A single read descriptor uses one rail, so single-rail and dual-rail results track together. This table shows the network limit for one operation, not the aggregate (the native figures above show the aggregate). Single rail, 16 threads. The rate climbs toward the single-rail line rate (~50 GB/s) as blocks grow and the fixed cost per operation is spread over more bytes:

Read latency: batching shares the round trip

A batch of read descriptors that share a block becomes one merged network operation, so the whole batch finishes in about the time of one unbatched read. Latency per descriptor therefore drops by close to the batch size: about 16× from batch_size=1 to batch_size=16 at the smaller block sizes. All 16 descriptors are served by one fetch and a local copy, instead of 16 separate round trips. Absolute latency per operation depends on how many threads share the rail, so the useful figure is the ratio: batching turns N descriptors into one network round trip.

Data verification (--check_consistency=1)

Throughput means nothing if the bytes are wrong, so the harness checks the data end to end with --check_consistency=1. This compares the actual bytes, not a checksum:

• Writes are read back from the drives. The harness fills the initiator buffer with a known value (0xaa) and writes it to MemKV. It then overwrites (poisons) that local buffer with 0x00, reads the data back from MemKV over RDMA, and compares every byte with the known value. A single wrong byte fails the run. This checks the whole path: client memory → server → NVMe → server → client memory.
• Reads check every delivered byte in the client buffer against the expected value.