vLLM + MemKV

vLLM moves KV cache through the LMCacheConnectorV1 connector, which hands each chunk to LMCache. LMCache loads storage backends as plugins, and the MemKV plugin is one of them. vLLM talks to LMCache, LMCache talks to the plugin, and the plugin talks to a MemKV cluster.

This page connects a vllm serve process to MemKV through LMCache, so KV chunks are stored in MemKV and read back.

Choose a vLLM path

vLLM can use MemKV in three ways:

The two native paths do not use LMCache and need vLLM 0.25.0 or later.

What you need

• A running MemKV cluster. See the Quick Start.
• A MemKV license file (minio.license).
• The MemKV auth key (32 bytes, hex-encoded).
• Network access to PyPI for the memkv-lmcache package.
• vLLM and LMCache. The official vllm/vllm-openai image includes vLLM; you install LMCache with pip.
• For the RDMA path, one or more RDMA NICs on the GPU host.

Step 1: install the plugin

The memkv-lmcache plugin is on PyPI for x86_64 and aarch64 Linux with glibc 2.28 or later. Install it with LMCache in the Python environment that vLLM runs in:

pip install 'lmcache>=0.5,<0.6' memkv-lmcache

In a container, install it when the container starts, as in the docker run example in Step 4.

Step 2: write the LMCache config

Create an LMCache YAML file that selects the MemKV backend:

chunk_size: 256
local_cpu: True
max_local_cpu_size: 60
enable_lazy_memory_allocator: True
lazy_memory_initial_ratio: 0.003
storage_plugins: memkv
extra_config:
  storage_plugin.memkv.module_path: memkv_lmcache.backend
  storage_plugin.memkv.class_name: MemKVStorageBackend

Two settings are required:

• local_cpu: True with max_local_cpu_size above 0. The plugin allocates retrieved tensors from LMCache's CPU pool.
• storage_plugins: memkv, with the extra_config entries that tell LMCache which module and class to load.

The other settings:

• max_local_cpu_size is the CPU pool size in GiB. A small pool (single-digit GiB) forces every read through MemKV. A very large pool can compete with CUDA for pinned host memory on systems that use it for NVLink transfers.
• enable_lazy_memory_allocator pins the CPU pool in 64 MiB chunks on a background thread, instead of pinning the whole pool at startup. Without it, a pool of several hundred GiB delays startup by tens of seconds, and the first read after a restart pays the same cost. It needs LMCache 0.5 or later, which is why the install command pins lmcache>=0.5,<0.6.
• lazy_memory_initial_ratio sets the part of the pool pinned at startup. Some drivers limit a single pinned allocation to 2 GiB; on those, keep ratio × max_local_cpu_size under 2 GiB (0.003 × 640 GiB ≈ 1.9 GiB for a 640 GiB pool). Where no such limit applies, you can raise it.
• chunk_size is the number of tokens LMCache writes at a time. Only prompts longer than chunk_size send traffic to MemKV, and each full chunk is one batched store call.

Step 3: configure the MemKV connection

The plugin reads the standard MemKV configuration: the MEMKV_CONFIG YAML file first, then MEMKV_* environment variables. Environment variables are usually enough:

export MEMKV_SERVERS="host-a:9900,host-b:9900"
export MEMKV_AUTH_KEY="<64-hex-char auth key>"
export MEMKV_TRANSPORT=auto          # or rdma; tcp only without RDMA
export MEMKV_LICENSE=/path/to/minio.license
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"     # required for the RDMA fast path
export MEMKV_STAGING_SIZE_MB=1024             # per-rank staging pool
export MEMKV_STAGING_SLOT_MB=16               # per-op staging slot

The staging settings set how much pinned host memory each tensor-parallel (TP) worker reserves. With TP=8, each host reserves 8 × MEMKV_STAGING_SIZE_MB.

Choose the transport

• auto (recommended) uses RDMA when MEMKV_RDMA_DEVICES names a NIC, and TCP otherwise.
• rdma uses RDMA only. If MEMKV_RDMA_DEVICES is not set, it uses mlx5_0 on Linux.
• tcp uses TCP only. Use it only when RDMA cannot reach the servers, for example in a container without /dev/infiniband or on a routed network without RoCE.

In Docker, RDMA needs --device=/dev/infiniband, --cap-add=IPC_LOCK, and --ulimit memlock=-1, as in the example below.

With tcp, the startup log still shows the RDMA device starting, but all KV transfers use TCP. To confirm the mode, look for transport: Auto or transport: Rdma in the Creating memkv engine line.

HMAC authenticates every message but does not encrypt it, so use TCP only on a network you control. See Encryption.

Step 4: launch vLLM with the LMCache connector

Pass LMCacheConnectorV1 in --kv-transfer-config, and point LMCache at the YAML file from Step 2 with LMCACHE_CONFIG_FILE:

docker run -d --name vllm-memkv \
    --runtime=nvidia --net=host --shm-size=64g --ipc=host \
    --device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1 \
    -e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    -e MEMKV_SERVERS="host-a:9900,host-b:9900" \
    -e MEMKV_AUTH_KEY="$AUTH_KEY" \
    -e MEMKV_TRANSPORT=auto \
    -e MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1" \
    -e MEMKV_LICENSE=/minio.license \
    -e LMCACHE_CONFIG_FILE=/lmcache.yaml \
    -v /path/to/models:/inference-models:ro \
    -v /path/to/minio.license:/minio.license:ro \
    -v /path/to/lmcache.yaml:/lmcache.yaml:ro \
    --entrypoint bash \
    vllm/vllm-openai:<tag> -lc '
      pip install "lmcache>=0.5,<0.6" memkv-lmcache &&
      vllm serve /inference-models/<model-dir> \
        --host 0.0.0.0 --port 8810 \
        --tensor-parallel-size <N> \
        --trust-remote-code \
        --max-model-len <max_seq_len> \
        --gpu-memory-utilization 0.85 \
        --enable-prefix-caching \
        --kv-transfer-config "{\"kv_connector\":\"LMCacheConnectorV1\",\"kv_role\":\"kv_both\"}"
    '

Check the startup log

A clean start logs these lines for each TP worker, in this order:

• Successfully installed memkv-lmcache-<version>: pip installed the plugin.
• LMCache INFO: Creating LMCacheEngine with config: {... storage_plugins: ['memkv'] ...}: LMCache read your YAML file.
• LMCache INFO: Created dynamic backend: memkv: LMCache loaded the plugin.
• INFO Creating memkv engine config=PluginConfig {...}: the plugin started its client.
• INFO memkv-client license verified plan=...: the license is valid.
• INFO Server mapped to rail server="...": the client registered a MemKV server. There is one line for each server.
• INFO MemKVStorageBackend ready (servers=[...], rdma=..., dst_device=...): the plugin is ready.

When the first prompt longer than chunk_size arrives, LMCache logs lines like:

LMCache INFO: [req_id=...] Stored ... tokens. ...
                        ... offload_time: ... put_time: ...

put_time is the time the plugin spent storing the chunk in MemKV.

What this integration gives you

• Capacity beyond host memory. When LMCache's CPU pool is full, evicted chunks go to MemKV instead of being dropped. The prefix cache becomes (host RAM × replicas) plus the MemKV cluster.
• Sharing across replicas. vLLM processes that point at the same MemKV cluster share one chunk pool. A system prompt used by many replicas is stored once.
• Durability. Stored chunks survive vLLM and LMCache restarts, and a restarted process reads them back.
• Authentication. Every operation is authenticated with HMAC and the cluster's shared key.

Operational notes

• Connections per rank. LMCache runs one cache engine per TP worker, and each has its own MemKV client. With TP=N, each MemKV server sees N sets of connections.
• The license is required. The plugin checks the license when it starts. Without a valid license, LMCache cannot start the backend and vLLM stops. Mount the license file and set MEMKV_LICENSE, or set it in the MEMKV_CONFIG file.

References

• Upstream vLLM project
• Upstream LMCache project
• LMCache storage_plugins extension docs