Architecture
The three ways to deploy MemKV, why its servers share nothing, and where the data lives.
MemKV connects to each inference engine through a plugin for that engine's storage interface. The Dynamo path uses NVIDIA NIXL, a library that moves data between GPU memory, host memory, and storage.
You can deploy MemKV in three shapes.
All three use the same wire protocol, the same authentication, and the same memkv-client engine.
Only the storage backend and the transport differ.
FIG. 01: Distributed (multi-server, RDMA and JBOF)
This is the high-performance shape, and the one benchmarked at 97.4 GB/s.
Several MemKV servers each store data on their own NVMe drives, used as JBOF (just a bunch of flash: raw drives with no RAID layer).
Data moves over RDMA with DC (Dynamically Connected) transport.
Each server writes to NVMe through io_uring + O_DIRECT and stages transfers in bounce buffers backed by hugepages.
FIG. 01·DR: Co-located with drives (single GPU node)
One MemKV server runs on the same host as the inference engine, with a few local NVMe drives.
It works the same way as the distributed shape (RDMA, DC, JBOF, and io_uring), but the data stays inside one chassis.
The local NIC moves bytes between memory regions without going through the IP stack.
FIG. 01·FL: Co-located, file mode (no raw drives)
Use this shape on hosts that have no raw drives to dedicate to MemKV: developer laptops, Apple Silicon Macs, edge AI nodes, and kind or CI clusters. MemKV stores data in files on a regular filesystem and uses TCP only. There is no RDMA, no JBOF, and no hugepages. Authentication, licensing, and the wire format are the same as in the other shapes. The host filesystem and NIC limit the bandwidth.
Supported hosts
The server runs on commodity x86 and ARM Linux hosts. The distributed shape needs NVMe drives and an mlx5 RDMA NIC. macOS and Apple Silicon support only the file-mode co-located shape.
Shared-nothing architecture
No gossip. No quorum. No rebalance. Every MemKV server is a self-contained unit. The client, not the servers, spreads data across the fleet.
MemKV servers do not talk to each other. There is no replication, leader election, or internal routing layer. Each server owns its local NVMe drives and serves them on its own. The client (the NIXL plugin or your application) spreads keys across servers and talks to each server directly.
This design is why throughput scales linearly:
- Client-side routing. Clients shard keys across servers and send RDMA traffic straight to the server that owns each key.
- No cross-server traffic. A new server adds capacity without adding coordination work, and nothing competes with the RDMA data path.
- Independent failures. A server failure affects only that server. For the blocks it owned, the client reroutes or falls back to computing the prefill from scratch.
- Simple operation. There is no quorum, no split-brain, and no cluster-wide configuration to keep in sync.
N servers deliver about N times the throughput of one server, because nothing between them can become a bottleneck.
See also
- Transport: the RDMA and TCP transports, authentication, and the offload flow.
- Storage: extent layout, on-drive versioning, and TRIM.
- Benchmarks: measured throughput and linear scaling.