What is MemKV?

Long contexts and many concurrent requests fill GPU high-bandwidth memory (HBM) with KV cache. When the cache no longer fits, the engine must run prefill again: the pass that computes the KV cache for every token in the prompt. Prefill then dominates each request, and throughput drops.

MemKV is a distributed store for KV cache. It keeps the cache on NVMe drives in a cluster of servers, so the engine reads a stored prefix back instead of computing it again. Throughput grows linearly as you add servers.

Under a production-shaped agentic workload, once the working set outgrows HBM, MemKV lets the same GPUs complete up to 3.4× the work and cuts continuing-turn tail latency up to 4.5×. Before that point, writing every block through to MemKV costs some throughput. The inference performance page has both results for eight models.

Supported inference engines

Each engine connects through its own storage interface:

• vLLM, unmodified, in three ways: behind LMCache, as a secondary tier under vLLM's native KV offloading, or as the offload medium itself. Only the third way moves blocks between the GPU and MemKV over RDMA with no CPU pool in between.
• sglang, unmodified, as a HiCache storage backend.
• NVIDIA Dynamo, through the same LMCache connector on a stock build, or through KVBM and NIXL, which needs a forked Dynamo build.

Key capabilities

• Zero-copy RDMA. Data moves directly between client host memory and NVMe over the DC transport. NICs other than Mellanox use the RC transport instead.
• RDMA control plane. On a host with an RDMA NIC, control messages also travel over RDMA. Only the first connection message uses TCP, because the RDMA connection does not exist yet.
• Full TCP transport. Where RDMA does not reach end to end, such as routed fabrics, cloud networks, or mixed NICs, MemKV runs the full data path over TCP. The client opens several connections to each server and spreads batched reads and writes across them.
• macOS support. MemKV runs beside an inference engine on a Mac, with file-mode storage and TCP, for development work.
• Authentication. Every message is signed with HMAC-SHA256 and a shared key. There is no unauthenticated mode.
• Extent-based block store. Large KV blocks are read and written in parallel.
• Linear scaling. Servers share nothing, so each server you add adds throughput.

Explore the docs

References

• NIXL — NVIDIA Inference Xfer Library
• Dynamo — NVIDIA Inference Framework