MinIO MemKV Documentation

Run NVIDIA Dynamo with MemKV as the remote KV cache tier behind KVBM, through the MemKV NIXL plugin over RDMA.

NVIDIA Dynamo runs vLLM behind its own DynamoConnector. Dynamo's KV block manager (KVBM) offloads evicted blocks through NIXL to a remote backend. MemKV provides a NIXL plugin, libplugin_MEMKV.so, that registers as that backend. With the plugin in the Dynamo container and DYN_KVBM_NIXL_BACKEND=MEMKV set, KVBM sends offloaded blocks to a MemKV cluster over RDMA.

Upstream Dynamo has no extension point for adding a remote KVBM backend, so this integration needs a forked Dynamo build that adds the MEMKV backend.

What you need

A running MemKV cluster. See the Quick Start.
A MemKV license file (minio.license).
The MemKV auth key (32 bytes, hex-encoded).
A Dynamo runtime image with KVBM and the MemKV backend, built from nvcr.io/nvidia/ai-dynamo/vllm-runtime.
libplugin_MEMKV.so for your platform.
One or more RDMA NICs on the GPU host.

Step 1: download the NIXL plugin

Download the plugin to the GPU host:

sudo mkdir -p /opt/memkv-plugins
sudo curl -L \
  https://dl.min.io/aistor/memkv/release/linux-amd64/libplugin_MEMKV.so \
  -o /opt/memkv-plugins/libplugin_MEMKV.so

If you installed MemKV from the .deb or .rpm package, the plugin is already at /usr/lib/memkv/libplugin_MEMKV.so.

In Step 3 you mount this file into the container at the path NIXL scans, /opt/nvidia/nvda_nixl/lib/plugins/libplugin_MEMKV.so. The plugin supports the NIXL memory types DRAM_SEG and VRAM_SEG. VRAM_SEG needs the NIC to register GPU memory, as a dma-buf or through a kernel peer-memory client: nvidia-peermem on NVIDIA, amdgpu's own on AMD.

Step 2: configure the MemKV connection

The plugin reads the standard MemKV configuration. Environment variables are the simplest way to set it:

export MEMKV_SERVERS="host-a:9900,host-b:9900"
export MEMKV_AUTH_KEY="<64-hex-char auth key>"
export MEMKV_TRANSPORT=auto              # auto | rdma | tcp
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"
export MEMKV_LICENSE=/path/to/minio.license
export MEMKV_STAGING_SIZE_MB=256
export MEMKV_STAGING_SLOT_MB=16
export MEMKV_NUM_CONNECTIONS=4

MEMKV_RDMA_DEVICES names the NICs the plugin uses. See Configuration → Client for every setting.

Step 3: launch Dynamo with KVBM pointed at MemKV

Compared with an ordinary vLLM launch, the Dynamo container needs:

The NIXL plugin mounted in NIXL's plugin directory.
DYN_KVBM_NIXL_BACKEND=MEMKV and DYN_KVBM_REMOTE_STORAGE_TYPE=memkv, so KVBM selects MemKV.
The libucx0 package. The NIXL stack the plugin links against needs the UCX runtime, which the image does not include.
For RDMA, --device=/dev/infiniband, --cap-add=IPC_LOCK, and --ulimit memlock=-1.

docker run -d --name dynamo-memkv-vllm \
    --runtime=nvidia --net=host --shm-size=64g --ipc=host \
    --device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1 \
    -e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    -e DYN_KVBM_NIXL_BACKEND=MEMKV \
    -e DYN_KVBM_REMOTE_STORAGE_TYPE=memkv \
    -e DYN_KVBM_CPU_CACHE_GB=60 \
    -e MEMKV_SERVERS="host-a:9900,host-b:9900" \
    -e MEMKV_AUTH_KEY="$AUTH_KEY" \
    -e MEMKV_TRANSPORT=auto \
    -e MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1" \
    -e MEMKV_NUM_CONNECTIONS=4 \
    -e MEMKV_STAGING_SIZE_MB=256 \
    -e MEMKV_STAGING_SLOT_MB=16 \
    -e MEMKV_LICENSE=/minio.license \
    -v /path/to/models:/inference-models:ro \
    -v /path/to/minio.license:/minio.license:ro \
    -v /opt/memkv-plugins/libplugin_MEMKV.so:/opt/nvidia/nvda_nixl/lib/plugins/libplugin_MEMKV.so:ro \
    nvcr.io/nvidia/ai-dynamo/vllm-runtime:<tag> bash -lc '
      apt-get update && apt-get install -y --no-install-recommends libucx0 &&
      dynamo serve /inference-models/<model-dir> \
        --host 0.0.0.0 --port 8810 \
        --tensor-parallel-size <N> \
        --enable-prefix-caching \
        --max-model-len <max_seq_len> \
        --gpu-memory-utilization 0.9
    '

Replace the image with your forked Dynamo build.

What this integration gives you

Capacity beyond GPU memory. KVBM moves evicted blocks from GPU high-bandwidth memory (HBM) into MemKV instead of dropping them. The KV cache becomes HBM plus the CPU pool plus the MemKV cluster.
Sharing across replicas. Dynamo workers that point at the same MemKV cluster share one block pool.
RDMA speed. With MEMKV_TRANSPORT=auto and RDMA NICs, blocks move at NIC line rate. The benchmarks page shows 97.4 GB/s on 2 servers, ~97% of 2× 400GbE.
Durability. Stored blocks survive Dynamo restarts.
Authentication. Every operation is authenticated with HMAC and the cluster's shared key.

Operational notes

The license is required. The plugin checks the license when NIXL loads it. Without a valid license, Dynamo cannot start the backend.
Connections per rank. Each tensor-parallel rank opens its own connections, so with TP=N each MemKV server sees N sets of connections from one worker.

Dynamo + MemKV