What is MemKV?

As inference models scale to longer contexts and higher concurrency, context memory becomes the bottleneck. GPU high-bandwidth memory (HBM) fills up, prefill — the pass that computes the KV cache for every token in the prompt — dominates the request, and throughput collapses. MemKV is a distributed shared context-memory store that bridges GPU memory and NVMe, scaling linearly as you add servers.

MemKV plugs into the engines that serve production traffic, each through the engine's own storage abstraction. vLLM connects three ways, all on a stock engine: behind LMCache, as a secondary tier under vLLM's native KV offloading, or as the offload medium itself — that third path alone moves blocks between the GPU tensors and MemKV over RDMA with no CPU pool in between. sglang attaches as a HiCache storage backend, likewise unmodified. NVIDIA Dynamo reaches MemKV two ways: through that same LMCache connector on a stock build, or through KVBM and NIXL, which needs a forked Dynamo until the MEMKV backend lands upstream.

Smaller deployments are covered too. For workstation- and laptop-class work — a Mac mini or Mac Studio, a single node running llama.cpp — MemKV also ships a vendor-neutral kv_store_v1 plugin that the inference server loads via dlopen, with no RDMA and no transfer-library glue. It is the enthusiast and single-box path rather than the production one, and it reads the same cluster the engines above write.

Key capabilities

• Zero-copy RDMA — Direct transfer between client host memory and NVMe via DC transport (RC fallback for non-Mellanox NICs)
• RDMA-native control plane — When the host has an RDMA NIC, control messages (Allocate, Lookup, Commit, Delete, Exists, Read, Write, BatchRead, BatchWrite) ride RC SEND/RECV on the per-connection queue pair. The bootstrap message (Connect) goes over a long-lived TCP connection, because the RC queue pair does not exist yet. Once it is up, control switches to RDMA.
• First-class TCP transport — RDMA is not always reachable end to end: routed or multi-hop fabrics, cloud, mixed NICs. MemKV runs the full data path over TCP for those, rather than degrading to a one-request-at-a-time fallback. The client opens a pool of connections per server and pipelines batched reads and writes across them, so one server's traffic spreads over many flows and sustains high throughput.
• macOS support — Runs co-located with an inference engine on a Mac, using file-mode storage and TCP, because RDMA, JBOF, and hugepages are not compiled in there. Useful for laptop and Mac mini development work.
• HMAC-SHA256 authentication — Every wire message is signed with a shared key; there is no unauthenticated mode
• Plugs into production inference engines — vLLM (via LMCache, native tiering, or direct offload), sglang HiCache, and Dynamo/KVBM over NIXL, each through the engine's own storage abstraction. A vendor-neutral kv_store_v1 plugin covers llama.cpp for single-box and enthusiast setups
• Extent-based block store — Parallelized I/O for large context blocks
• Linear scalability — Shared-nothing architecture; add servers to scale throughput

Why MemKV?

Explore the docs

References

• llama.cpp — upstream inference server; v2 chunked slot save fork lives at minio/llama.cpp
• NIXL — NVIDIA Inference Xfer Library
• Dynamo — NVIDIA Inference Framework