MinIO MemKV Documentation

Add MemKV as a secondary tier under vLLM's native KV offloading, so blocks evicted from the CPU pool come back from MemKV instead of being recomputed.

vLLM's native KV offloading (OffloadingConnector) copies finished KV blocks from the GPU into a pinned CPU pool. Its tiering mode (TieringOffloadingSpec) passes those blocks on to secondary tiers behind the pool. The memkv tier makes a MemKV cluster one of those tiers. When a block has left the CPU pool, vLLM reads it back from MemKV instead of computing it again, also after a restart and from other vLLM instances.

This path does not use LMCache. It keeps a pinned CPU pool in front of MemKV, which serves hits without contacting the cluster. To move blocks between the GPU and MemKV with no pool in between, see vLLM direct KV offloading + MemKV. For a comparison of all three vLLM paths, see Choose a vLLM path.

What you need

A running MemKV cluster. See the Quick Start.
A MemKV license file (minio.license).
The MemKV auth key (32 bytes, hex-encoded).
The memkv-vllm package. It runs on Linux only.
vLLM 0.25.0 or later. The plugin is tested on v0.25.0, v0.25.1, v0.26.0, v0.27.0, and v0.27.1. vLLM marks its offloading API as experimental and has changed it between releases, so test a new vLLM version before you deploy it.
For the RDMA path, one or more RDMA NICs on the GPU host. Without RDMA, the tier uses TCP.
PYTHONHASHSEED set to the same value, for example 0, on every vLLM instance that should share KV cache. Otherwise vLLM seeds block hashes differently in each process, and lookups from another instance, or from the same instance after a restart, miss.

Step 1: install the plugin

pip install memkv-vllm   # or a local wheel: pip install memkv_vllm-*.whl

The package registers itself with vLLM as a general plugin, so vLLM needs no patch and no extra flag. If you limit plugins with VLLM_PLUGINS, add memkv_tier to the list. The plugin does nothing until you add a memkv tier to the serve configuration.

Step 2: point the client at MemKV

The tier uses the standard MEMKV_* environment variables, like the other MemKV plugins:

export MEMKV_SERVERS=10.0.0.17:9900
export MEMKV_AUTH_KEY=<64-hex-char key>
export MEMKV_LICENSE=/etc/memkv/license      # JWT or path
export MEMKV_RDMA_DEVICES=mlx5_1             # optional, RDMA path

You can instead mount a client.yaml file and set MEMKV_CONFIG=/path/client.yaml.

At startup the tier contacts every configured server and stops with an error if one is wrong. This prevents a bad server entry from losing its share of the stored blocks later.

Step 3: serve with the tiering connector

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 17179869184,
      "block_size": 64,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'

block_size is in tokens and must be a multiple of the GPU block size. It sets the size of each MemKV value: one value holds one offloaded block across all tensor-parallel ranks. Choose it so values are several MiB; 64 tokens is 16 MiB for a Qwen2.5-32B fp16 KV layout. Small values make reads slow because each key has a fixed cost.
cpu_bytes_to_use is the size of the CPU pool between the GPU and MemKV. Blocks are written to MemKV as soon as they are stored, and read from MemKV when a lookup misses the pool. A pool smaller than your active working set uses MemKV all the time. A large pool makes MemKV mainly the layer for restarts and for sharing between instances.

Tier settings

Set these in the memkv entry of secondary_tiers:

Setting	Default	Meaning
`type`		Must be `memkv`.
`prefix`	`""`	Key namespace.
`n_read_threads`	8	Threads for loads from MemKV.
`n_write_threads`	8	Threads for stores to MemKV.
`blocks_per_op`	8	Blocks per batched transfer.
`max_store_backlog_mb`	2048	Pending store data, in MiB, at which the tier skips new stores. Skipped blocks are not stored in MemKV; inference is not affected. Keep it below half of `cpu_bytes_to_use`.
`load_failure_grace_s`	60	How long a failed load waits before it is reported, so that a transfer abandoned by a client timeout cannot write into slots vLLM has reused. Leave it set. It costs nothing when loads succeed.
`lookup_timeout_s`	5.0	Deadline for a lookup. Needs a vLLM patch; see Bound your lookups.

Check the startup log

A clean start logs:

memkv-client license verified plan=ENTERPRISE-PLUS
MemKV tier ready (servers=[...], rdma=True, block_bytes=16777216, ...)
Created secondary tier #0 (memkv)
Created TieringOffloadingManager with primary tier (lru, 1024 blocks) and 1 secondary tier(s)

What this integration gives you

Prefixes survive CPU pool eviction. When a long context comes back, vLLM reads it from MemKV instead of computing it again.
Reuse across restarts and instances. Keys are content hashes, grouped by model and layout. A restarted engine, or a second instance with the same configuration and the same PYTHONHASHSEED, finds the blocks stored before.
RDMA end to end. The whole CPU pool is registered for RDMA once at startup, and loads arrive by RDMA directly in the destination slots.
Failures do not block inference. If MemKV is unreachable, lookups report a miss and vLLM computes the block again. If a block is evicted between lookup and load, that load fails and the block is computed again.

Operational notes

Where the tier runs. The tier runs in vLLM's scheduler (EngineCore) process, with one MemKV client for each vLLM instance, not for each GPU worker. That process needs the RDMA libraries (libibverbs, ibverbs-providers) and access to the RDMA device. Its locked-memory limit (RLIMIT_MEMLOCK) must be at least cpu_bytes_to_use, because the whole pool is pinned as one region. In a container, grant IPC_LOCK.
Metrics. vLLM's Prometheus endpoint reports the tier under vllm:kv_offload_memkv_* (bytes stored and loaded, job latency, failures, dropped stores, and lookup results), beside the connector's own vllm:kv_offload_* series. To export the MemKV client's transport metrics, set MEMKV_PROMETHEUS_BIND.
Capacity. MemKV decides what to evict. There is no eviction setting on the vLLM side.

Bound your lookups (patch required)

vLLM's lookup manager has no deadline. It waits until each lookup returns, and holds the request back meanwhile. One slow lookup, for example behind a burst of stores or after a server restart, can then stop the whole engine instead of falling back to recomputing. We measured 62 of 64 requests held back with zero completions for a 12-minute window.

The fix gives every lookup a deadline. A lookup that cannot answer in time counts as a miss, and the block is computed again. The fix has two parts:

In the plugin, from the memkv-vllm package in RELEASE.2026-08-04T22-19-45Z (PyPI 1.0.5) and later: the lookup_timeout_s tier setting, and the vllm:kv_offload_memkv_lookups{result="timeout"} counter, which shows slow lookups before they stop the engine.
In vLLM: a patch to the engine's lookup manager that enforces the deadline. No vLLM release includes it yet.

Contact MinIO support or your account team for the vLLM patch. On a vLLM without the patch, the plugin logs a warning, ignores lookup_timeout_s, and lookups have no deadline.

After you apply the patch, set the deadline in the tier entry:

{"type": "memkv", "lookup_timeout_s": 5.0}

The default is 5 seconds. A value of zero or less turns the deadline off.

References

vLLM KV offloading guide
Tiering framework — vllm-project/vllm#40020
MemKV LMCache path: vLLM + MemKV

vLLM native offloading + MemKV