vLLM native offloading + MemKV

vLLM's native KV offloading (OffloadingConnector) copies finished KV blocks from GPU into a pinned CPU pool as they are produced, and its tiering mode (TieringOffloadingSpec) cascades those blocks to secondary tiers behind that pool. The memkv tier makes a MemKV cluster one of those tiers: blocks evicted from the CPU pool are promoted back from MemKV instead of being recomputed, across engine restarts and across instances. vLLM never sees MemKV directly — the scheduler talks to its tiering manager, the manager talks to the tier, and the tier moves bytes between CPU pool slots and MemKV.

This is a different integration than the LMCache path: no LMCache anywhere, vLLM's own offloading framework end to end. The rest of this page is the wire-up: get a vLLM serve invocation cascading KV blocks into MemKV and promoting them back on demand.

What you need

• A running MemKV cluster (one or more nodes).
• A MemKV license file (minio.license).
• The MemKV auth key (32-byte HMAC, hex-encoded).
• The memkv_vllm wheel (Linux only — the data path is not compiled for other platforms).
• vLLM main at or after commit 6cf7b26bd (v0.23.1rc1.dev962). The tiering framework has existed since v0.22.0, but the secondary-tier API this plugin implements is newer than the latest stable release (v0.24.0) — it first appears in the v0.25.0 release candidates; upstream marks the whole offloading spec API experimental. Use vLLM's per-commit wheels (https://wheels.vllm.ai/<sha>/) or postmerge CI images until a stable release ships the current surface.
• One or more RDMA NICs visible on the GPU host if you want the fast path — set MEMKV_RDMA_DEVICES=mlx5_0,mlx5_1 to bind them. Without RDMA the tier runs over TCP.
• PYTHONHASHSEED pinned (for example 0) on every vLLM instance that should share KV. vLLM seeds block content hashes per process otherwise, and lookups from another instance (or the same one after a restart) silently miss.

Step 1: install the plugin

pip install memkv-vllm   # or a local wheel: pip install memkv_vllm-*.whl

The wheel registers itself through the vllm.general_plugins entry point — no vLLM patches, no extra flags. If you restrict plugins with VLLM_PLUGINS, include memkv_tier in the list. Installing the wheel changes nothing until a memkv tier appears in the serve configuration below; registration is lazy and adds no imports to engines that do not use it.

Step 2: point the client at MemKV

The connection uses the standard MEMKV_* env chain, identical to the other MemKV plugins:

export MEMKV_SERVERS=10.0.0.17:9900
export MEMKV_AUTH_KEY=<64-hex-char key>
export MEMKV_LICENSE=/etc/memkv/license      # JWT or path
export MEMKV_RDMA_DEVICES=mlx5_1             # optional, RDMA path

Or mount a client.yaml and set MEMKV_CONFIG=/path/client.yaml. The tier probes every configured server at startup and fails loudly on misconfiguration — with HRW sharding, a bad server entry that passed startup would silently lose its share of the keyspace at runtime instead.

Step 3: serve with the tiering connector

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "TieringOffloadingSpec",
      "cpu_bytes_to_use": 17179869184,
      "block_size": 64,
      "secondary_tiers": [
        {"type": "memkv", "n_read_threads": 8, "n_write_threads": 8}
      ]
    }
  }'

block_size (tokens, a multiple of the GPU block size) sets the MemKV value size: one value holds one offloaded block across all tensor-parallel ranks. Size it so values land in the megabytes — 64 tokens is 16 MiB for a Qwen2.5-32B fp16 KV layout — because small values make reads per-key-overhead-bound rather than bandwidth-bound.

cpu_bytes_to_use is the CPU pool between GPU and MemKV. MemKV sees traffic when blocks cascade (immediately, on store) and when a lookup misses the pool (promotion). A pool smaller than your hot working set exercises MemKV constantly; a big pool makes MemKV the restart-survival and cross-instance layer.

Tier keys beyond type: prefix (key namespace), blocks_per_op (blocks per batched transfer, default 8), load_failure_grace_s (see operational notes), max_store_backlog_mb (backpressure shed threshold, default 2048 — keep below half of cpu_bytes_to_use).

Clean startup logs look like:

memkv-client license verified plan=ENTERPRISE-PLUS
MemKV tier ready (servers=[...], rdma=True, block_bytes=16777216, ...)
Created secondary tier #0 (memkv)
Created TieringOffloadingManager with primary tier (lru, 1024 blocks) and 1 secondary tier(s)

What this integration buys

• Prefixes survive CPU-pool eviction — the pool is finite; MemKV is the tier behind it. Re-visited long contexts promote back instead of recomputing.
• Restart and cross-instance reuse — keys are content hashes namespaced by model and layout, so a restarted engine (or a second instance with the same configuration and pinned PYTHONHASHSEED) hits blocks its predecessor stored.
• RDMA end to end — stores publish CPU pool slot addresses to the client; loads land via server-driven RDMA WRITE into the destination slots. The CPU pool is pre-registered as one region at startup, so per-transfer registration never fires.
• Failure isolation — a MemKV outage degrades lookups to MISS (recompute); a value evicted between lookup and load fails that promotion and the block recomputes. Inference never blocks on the tier.

Operational notes

• The tier runs in the scheduler (EngineCore) process: one MemKV client per vLLM instance, not per GPU worker. That process needs the RDMA userspace libraries (libibverbs, ibverbs-providers) and device access, and its RLIMIT_MEMLOCK must accommodate cpu_bytes_to_use — the whole pool is registered as one pinned region (grant IPC_LOCK, as the pod pattern does).
• Metrics surface on vLLM's Prometheus endpoint under vllm:kv_offload_memkv_* (store/load bytes, job latency by op, failures, sheds, lookup results), next to the connector's own vllm:kv_offload_* series. Client transport metrics are available separately via MEMKV_PROMETHEUS_BIND.
• load_failure_grace_s (default 60) withholds a failed load's completion so a transfer abandoned by a client timeout cannot race the framework's reuse of the destination slots. Leave it on; it costs nothing on the happy path.
• Failed loads purge their keys, so a value that cannot load (for example stored by an instance with an incompatible slot stride) recomputes and re-stores once instead of looping.
• Capacity is governed by MemKV lane eviction, not the tier; there is no tier-side eviction bookkeeping to configure.
• The upstream offloading API is experimental and has already changed between vLLM releases. The plugin pins a supported range; re-qualify with the conformance suite when moving vLLM versions.

Roadmap

• Engine-owned staging with quarantine for abandoned transfers, replacing the grace-period mitigation.
• Tuning defaults from the win-regime benchmark on H100-class hardware, plus head-to-head numbers against vLLM's fs and obj tiers.
• Upstream: a module_path fallback for out-of-tree tiers (prepared in vllm-plugin/upstream/).

References

• vLLM KV offloading guide: https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html
• Tiering framework: vllm-project/vllm#40020
• MemKV LMCache path (the stable-vLLM alternative): vLLM + MemKV