Benchmarks

This page measures the wire: how fast bytes move between a client and MemKV. For what that buys an inference server under a production-shaped agentic workload — continuing-turn TTFT, tok/s/GPU, and the below-cliff cost — see Inference performance.

Throughput by block size (2-server aggregate)

Test harness. 1 GPU node (2× ConnectX-7 400GbE) driving 2 storage servers (12 NVMe each) through the memkv bench binary at 64 threads per server (128 total on the dual-server sweep). RDMA and TCP were swept separately. Last verified against main on 2026-07-06.

RDMA peak: 97.4 GB/s read, 97.4 GB/s write — ~97% of 2× 400GbE line rate. TCP peak: 22.3 GB/s read, 16.3 GB/s write. RDMA tracks near line rate across the sweep. TCP remains a viable transport wherever RDMA can't reach end to end (routed/multi-hop fabrics, cloud, mixed NICs), sustaining tens of GB/s via multiplexed pipelining. TCP figures are a separate sweep over the inline-bulk path with 64 client connections (one per thread).

Linear scaling

Each server has 12 NVMe drives attached to the same PCIe domain as the NIC. No coordination or drive sharing between servers — add servers to scale throughput linearly.

Under live engine load

Wire sweeps above isolate the store. During the August 2026 NVIDIA AIPerf agentic campaign — the same 198 windows on 8× H200 / 2 MemKV nodes (48 raw NVMe, dual 400 Gb rails) that feed Inference performance — the storage tier was never the bottleneck:

NIXL / nixlbench on KV-shaped blocks over dual-rail RoCE sustained 104 GB/s in that campaign. That figure is the webinar transport headline; the batched-write table below is a separate nixlbench WRITE sweep (to 64 MiB, --recreate_xfer), and the 97.4 GB/s peak above is native memkv bench.

NIXL plugin (nixlbench)

NVIDIA's nixlbench (from nixl v1.2.0) driving the MemKV NIXL plugin against the same dual-server fleet, DRAM destinations, with --recreate_xfer (a fresh transfer every iteration — no cached-handle fast path), swept to nixlbench's default 64 MiB max block size. The plugin coalesces each post_xfer descriptor list inside the call: same-key contiguous-offset descriptors fold into one wide BatchRead, and the server's 2 MiB bounce-buffer chunking handles the actual transfer.

Batched write throughput (GB/s)

nixlbench WRITE, batch_size=16, --recreate_xfer, 64 threads, swept to the default 64 MiB max block size:

A batch's writes persist concurrently across the NVMe drives, so batched WRITE scales with block size — at 64 MB the single rail reaches ~89% of its line rate. These are nixlbench-driven figures; the native memkv bench aggregate (97.4 GB/s) is at the top of this page. The August 2026 webinar also quotes 104 GB/s sustained nixlbench on KV-shaped blocks over dual-rail RoCE — see Under live engine load.

Aggregate read throughput scales across rails exactly like writes — the top-of-page figures (48.7 GB/s single server, 97.4 GB/s dual) are RDMA reads. The plugin path adds one thing on top: shared-range coalescing.

Read coalescing — fetch once, scatter locally

The plugin coalesces a read descriptor list before it touches the wire. When several descriptors in one post_xfer resolve to the same KV block with overlapping or contiguous ranges, the optimizer picks one destination that spans the full range as a cover, issues a single RDMA read into it, then fills every other destination with a local copy from that cover — std::memcpy for host (DRAM) buffers, cudaMemcpy for GPU (VRAM) buffers via the dynamically-loaded CUDA runtime. The shared block crosses the NIC once and fans out to N buffers at local memory bandwidth (DRAM/HBM), not network bandwidth.

This is the prefix-sharing win: when many concurrent sequences read the same prompt prefix, it is pulled across the fabric once and scattered to every consumer in memory. (When destinations are GPU memory and no CUDA copy path is available, or no single destination covers the range, the optimizer splits the chunk back into independent per-destination wire ops — correctness first.)

Writes never coalesce this way — overwriting an object range from two source buffers is undefined — so each write persists independently and write throughput reflects true wire/NVMe bandwidth (above).

Why read throughput can exceed line rate

If you benchmark the plugin with nixlbench you will see read throughput reported well above NIC line rate — e.g. ~150 GB/s on a single 400GbE rail, roughly 3× its ~50 GB/s wire ceiling, at batch_size=16. This is expected: it is the coalescing optimizer showing through the benchmark's byte accounting, not a measurement error.

• nixlbench credits every descriptor in a batch with a full block transfer — it counts logical bytes delivered to the caller.
• The plugin keys each descriptor by its NIXL dev_id, and nixlbench's synthetic workload uses very few distinct keys (one when num_initiator_dev=1). A batch's descriptors therefore resolve to the same KV block, and the optimizer fetches that block once over the wire and scatters it to every destination locally.
• Reported throughput is logical bytes ÷ time. The shared bytes cross the NIC once but are counted N times, so the effective rate runs above the wire ceiling.

The bytes are real — every destination receives correct data (--check_consistency=1 passes), and the win is genuine for prefix-sharing reads. But it is an effective rate, bounded by local memory bandwidth for the shared portion, not raw network bandwidth. A workload of all-distinct keys sees no coalescing and reads track the wire-limited rate below.

Per-op read wire rate (batch_size=1)

A lone read descriptor uses one rail, so single- and dual-rail track together — this is the per-op wire ceiling, not the aggregate (that's the native figure above). Single rail, 16 threads; it climbs toward the single-rail line rate (~50 GB/s) as the block grows and per-op overhead amortizes:

Read latency — batching amortizes the round-trip

A batch of read descriptors that share a block collapses into one coalesced wire op, so the whole batch completes in roughly the wall-clock of a single un-batched read. Per descriptor, latency therefore drops by close to the batch factor — measured at about 16× from batch_size=1 to batch_size=16 at the smaller block sizes — because all 16 descriptors are served by that one fetch plus a local scatter, not 16 independent round-trips. Absolute per-op latency depends on concurrency (threads contending for the rail), so the figure that travels is the ratio: batching turns N descriptors into one wire round-trip.

Data verification (--check_consistency=1)

Throughput is meaningless if the bytes are wrong, so we verify data integrity with --check_consistency=1 — actual end-to-end verification, not a checksum stand-in:

• Writes are read back off the drives. The initiator buffer is filled with a known sentinel (0xaa) and written to MemKV. The harness then poisons that local buffer to 0x00, issues an RDMA read-back from MemKV, and byte-compares the returned data against the sentinel — a single wrong byte fails the run. This walks the whole path: client memory → server → NVMe → server → client memory.
• Reads verify every delivered byte in the client buffer against the expected sentinel.