vLLM direct KV offloading + MemKV

MemKVOffloadingSpec makes a MemKV cluster the offload medium for vLLM's native KV offloading. The worker is handed the live GPU KV tensors, packs each block into one value, and MemKV servers move it with one-sided RDMA. vLLM's own CPU pool is not in the path.

This is the third way to connect vLLM to MemKV:

The spec stages each transfer through a small pool of registered buffers. Those buffers can live in pinned host memory (the default) or in GPU memory, and that choice is the one decision this page is really about — it is worth up to 3.8x on restores, in either direction, depending on the machine.

What you need

• A running MemKV cluster.

• A MemKV license file and the 32-byte HMAC auth key, hex-encoded.

• The memkv_vllm wheel. Linux only — the data path is not compiled for other platforms.

• An RDMA NIC on the GPU host. This path has no TCP fallback. The worker fails at startup if the client cannot reach a NIC, rather than silently running slowly.

• A GPU and driver that can export device memory to the NIC, either through dmabuf or through nvidia-peermem. The client tries dmabuf first and falls back on its own.

• A vLLM build whose offloading API exposes OffloadingSpec and loads a spec from a module path. Check before you serve:

  python -c "from vllm.v1.kv_offload.base import OffloadingSpec, CanonicalKVCaches; print('ok')"

  If that import fails, your vLLM predates the API this plugin implements. Upstream marks the whole offloading spec API experimental, so re-check it when you move vLLM versions.

• PYTHONHASHSEED pinned to the same value on every instance that should share KV, for example 0. vLLM seeds block content hashes per process otherwise, and lookups from another instance — or the same one after a restart — miss without saying why.

Step 1: install the plugin

pip install memkv-vllm   # or a local wheel: pip install memkv_vllm-*.whl

Step 2: point the client at MemKV

The connection uses the standard MEMKV_* env chain, the same one every other MemKV plugin uses:

export MEMKV_SERVERS=10.0.0.17:9900
export MEMKV_AUTH_KEY=<64-hex-char key>
export MEMKV_LICENSE=/etc/memkv/license      # JWT or path
export MEMKV_RDMA_DEVICES=mlx5_1             # required on this path

You can mount a client.yaml and set MEMKV_CONFIG=/path/client.yaml instead.

Step 3: serve with the spec

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "MemKVOffloadingSpec",
      "spec_module_path": "memkv_vllm.spec"
    }
  }'

Do not set block_size in kv_connector_extra_config. This version requires one offload key per GPU block, and the worker asserts that at startup rather than storing a layout it cannot read back.

A healthy worker logs one line per rank:

MemKV GPUDirect worker: rank=0 tensors=80 layers/block=80 value=0.62MiB
  threads=4 chunk=64 blocks scratch=160MiB HBM

Those five numbers are what the next section is about.

Choose where KV stages

"memkv_scratch_medium": "host"    // default; or "gpu"

Both media move the same bytes under the same keys, so a cluster written under one is readable under the other. The difference is the path the NIC takes to reach the staging buffer:

• host — the NIC reads and writes ordinary pinned host memory, a path every server platform is built for. The GPU gather still runs on the GPU; its result crosses PCIe once into the buffer. Costs that one crossing, and does not care where the NIC sits.
• gpu — the NIC reads and writes HBM directly, so nothing crosses PCIe on the way to the wire. Faster only when the NIC and that rank's GPU sit on the same PCIe switch.

That condition is the whole story. Peer-to-peer between a NIC and a GPU on different switches has to climb to the CPU root complex and back, which is slow on current platforms. Run nvidia-smi topo -m and read the GPU-to-NIC entries: PIX means one switch, NODE and SYS mean the traffic crosses the root complex.

Set gpu when every rank has a PIX NIC — that is what rail-optimised hardware is for, and it pairs one NIC per GPU under a shared switch precisely so this holds. Keep the default host otherwise. A host with more GPUs than NICs cannot give most ranks a PIX NIC, whatever the plugin is set to.

memkv bench --auth-key <key> \
  --servers <rail0>:9900,<rail1>:9900 --devices mlx5_0,mlx5_1 \
  --vllm=host --kv-model llama-3.1-70b --tp 8 --gpu-block-size 64 \
  --concurrency 32 --verify

Size the staging pool

A KV block's layers are not next to each other in GPU memory, so the plugin gathers each block into one contiguous buffer and stores it as a single value. Those buffers are the staging pool. Under the default host medium they cost pinned host RAM; under gpu they cost VRAM the KV cache would otherwise have, which is what the rest of this section sizes.

The fastest way: read it from the log

The startup line above already tells you the answer:

• value= is the value length: one block across all layers, for one rank.
• chunk= is how many blocks share one scratch buffer.
• scratch= is the total the pool costs on this GPU.

Serve once, read the line, adjust if you need to. Everything below explains how those numbers arise, so you can plan before you serve.

The formula

Two settings control the pool, both under kv_connector_extra_config:

The value length comes from the model, not from you:

value_len = num_layers × 2 × block_size × (num_kv_heads ÷ TP) × head_dim × bytes_per_elem
            │             │        │              │                │            └── 2 for BF16/FP16
            │             │        │              │                └────────────── e.g. 128
            │             │        │              └─────────────────────────────── after GQA reduction
            │             │        └──────────────────────────────────────────── vLLM GPU block, in tokens
            │             └───────────────────────────────────────────────────── K and V stored separately
            └─────────────────────────────────────────────────────────────────── e.g. 80

The pool then follows in two steps:

chunk_blocks = clamp(floor(scratch_mb_per_thread × 1 MiB ÷ value_len), 1, 64)

scratch_total = num_threads × chunk_blocks × value_len

Which term binds: the cap or the budget

The 64-block cap and the MiB budget bind in different regimes, and the difference is large enough to matter:

Read it this way:

• Small blocks are cap-bound. Llama-3.1-70B reaches 64 blocks with only 40 MiB of its 128 MiB budget. Raising the budget buys nothing; the cap is already reached.
• Large blocks are budget-bound. GLM-5.2 spreads a block across 99 KV tensors, which makes one value about 3.5 MiB, so the budget runs out at 36 blocks. Here the budget is the dial, and lowering it lowers both the chunk size and the VRAM cost — cutting it to 48 MiB gives 13 blocks and 182 MiB.

How much can it ever use

Per GPU, the pool never exceeds:

scratch_total ≤ num_threads × max(scratch_mb_per_thread, value_len)

At the defaults that is 512 MiB per GPU, and every layout reaches it from below: 64 MiB for a small-block model, 160 MiB for Llama-3.1-70B, 504 MiB for GLM-5.2. Multiply by nothing for tensor parallelism — each rank has its own GPU and its own pool, so 512 MiB is the figure that competes with that GPU's KV cache.

Where this sits in the engine's memory budget

This subsection and the two after it apply to memkv_scratch_medium: gpu. Under the default host medium the pool costs pinned host RAM, competes with nothing on the GPU, and gpu_memory_utilization needs no adjustment at all.

A GPU running inference has its VRAM already committed. Per GPU:

total VRAM = model weights (this rank's shard)
           + activation and compute workspace peak   ← vLLM measures this
           + KV cache                                ← vLLM sizes this to fill
           │                                           gpu_memory_utilization
           + MemKV scratch pool                      ← this plugin
           + CUDA context, driver, fragmentation     ← outside the utilization
                                                       fraction

The order those happen in is what matters, because the plugin is last:

1. vLLM loads the model weights.
2. vLLM profiles a forward pass to find the activation peak.
3. vLLM sizes the KV cache to fill gpu_memory_utilization, then allocates it.
4. Only then does this plugin allocate its scratch pool, because the worker is handed the KV tensors after they exist.

So the pool is not inside the budget vLLM computed. It takes whatever VRAM is still free below gpu_memory_utilization, and if that is exhausted it takes from the margin you left for the driver and fragmentation — or fails to allocate and takes the engine down at startup.

Leave room in gpu_memory_utilization

vLLM will not make room for the pool, so you have to:

headroom = scratch_total ÷ total_gpu_memory

For 504 MiB of scratch on a 141 GB GPU that is 0.004, so --gpu-memory-utilization 0.90 becomes 0.89. Round down: a spare 100 MiB is cheaper than a failed start.

To avoid redoing this per model, reserve the ceiling once — 512 MiB at the defaults, 0.38% of a 141 GB GPU — and leave it alone.

What the KV cache gives up

Lowering gpu_memory_utilization shrinks the KV cache by exactly the amount you reserve, so the pool's real cost is measured in cached tokens. Divide by the per-token KV size for one rank:

tokens given up = scratch_total ÷ (value_len ÷ block_size)

For Llama-3.1-70B BF16 at TP8, one rank holds 40 KiB per token, so:

That is roughly one long request's worth of cache, traded for the ability to serve prefixes from MemKV across restarts and instances. Whether the trade pays depends on your reuse rate, not on the size of the pool.

The gather and scatter kernels also allocate short-lived temporaries, one layer's slice of a chunk at a time. That is scratch_total ÷ (num_threads × num_layers) per thread, so scratch_total ÷ num_layers across the whole worker — about 1% on top of the pool. For the 160 MiB Llama-3.1-70B pool that is 0.5 MiB per thread and 2 MiB in total. They live on the transfer threads' own CUDA streams, so PyTorch's allocator keeps them in a pool the compute stream will not reuse. Reserve for the pool and this is inside the rounding.

Size it to the wire, not to the budget

The pool exists to keep bytes on the wire. batch_xfer blocks until the server acknowledges a group, so each thread has exactly one chunk in flight, and the bytes in flight for the whole worker are:

in-flight bytes = num_threads × chunk_blocks × value_len = scratch_total

Saturating a link needs in-flight bytes equal to its bandwidth-delay product — bandwidth multiplied by the round trip of one group. Above that, more scratch buys no throughput at all; it only takes KV cache away from the engine.

useful scratch_total ≈ achievable_bandwidth × round_trip_per_group

The catch is that the round trip is not a constant you can look up. A job cannot finish faster than its own bytes take to move, so a bigger chunk lengthens the round trip that justifies it. The product has to be measured, not assumed.

memkv bench --vllm measures it, moving KV in the plugin's shape with the cache in GPU memory:

memkv bench --auth-key <key> \
  --servers <rail0-addr>:9900,<rail1-addr>:9900 \
  --devices mlx5_0,mlx5_1 \
  --vllm --kv-model llama-3.1-70b --tp 8 --gpu-block-size 16 \
  --gpu-devices 0,4 --concurrency 1,2,4

It sweeps concurrency against the plugin's scratch budgets, reports KV tokens per second, and recommends the smallest footprint that saturates — along with what that costs in KV cache tokens.

Give each rail a server address on that rail's own subnet. A rail talking to an address on another rail's subnet stalls rather than saturates.

What gpu costs without PCIe affinity

Choosing the medium turns on whether each rank has a NIC on its own PCIe switch. This is what the answer is worth, measured on one 400 Gb rail restoring 2.5 MiB blocks at 32 concurrent jobs, changing nothing but which GPUs the traffic came from:

Note what that last row costs: the rail delivers a third of its line rate, not merely less per GPU.

This is topology, not tuning. On the measured host each GPU sits under its own PCIe switch and each NIC shares a switch with exactly one of them, so every other GPU's traffic crosses switches through the CPU root complex. We tested the usual suspect — PCIe ACS, which NVIDIA asks you to disable for direct GPU transfers because its redirect bits push peer-to-peer traffic up to the root complex — by clearing ReqRedir, CmpltRedir and UpstreamFwd on every switch port and re-running: 17.5 GB/s with ACS enforcing, 18.2 GB/s with it cleared. ACS is worth disabling on general principle, but it is not the lever here.

So a host with fewer NICs than GPUs cannot reach line rate on every rank, however the plugin is tuned. For full throughput give each GPU a NIC on its own PCIe switch, which is how rail-optimised nodes are built. Short of that, leave memkv_scratch_medium at host: it sidesteps the question entirely and, on the machine above, restored 3.8x faster than gpu did.

Do not carry those numbers to another host. Fabric, drive latency, and how many ranks share a rail all move the product, and a TP=8 deployment puts eight ranks on the rails a one-rank measurement had to itself.

What the pool costs, by medium

Under host the pool is pinned host RAM — a few hundred MiB on a machine that has hundreds of gigabytes, competing with nothing the engine needs.

Under gpu it is VRAM the KV cache would otherwise hold, and the sections above price that: 512 MiB at the defaults, about 13,100 cached tokens per GPU for Llama-3.1-70B at TP8. That is the price of the direct path, and it is worth paying only where the direct path is actually faster — which the medium section reduces to one question about your PCIe layout.

This is a different question from whether to use a DRAM cache tier. The pool here is staging, not residency: it holds a chunk in flight and nothing else. If you want host memory serving hits without touching the cluster, that is native tiering, which puts vLLM's own pinned pool in front of MemKV and is a separate integration.

Choosing the two settings

Start with the defaults. Change them only for a reason:

• VRAM is tight. Lower memkv_scratch_mb_per_thread first. It costs transfer efficiency, because a smaller chunk means more RDMA round trips for the same blocks.
• Transfers are not keeping up. Raise memkv_num_threads. VRAM grows in proportion, since every thread gets its own buffer.
• Never let the chunk fall to 1 block. That happens when the budget is smaller than one value length, and it removes batching from the transfer path entirely. If the startup log shows chunk=1 blocks, raise the budget above value=.

What this integration buys

• Blocks move between MemKV and the GPU tensors, under either medium. The worker holds the live KV tensors, so it gathers out of them and scatters back into them itself. The staging buffer carries one chunk in flight and nothing else — it is transit, not a tier, and memkv_scratch_medium only decides where that transit happens. Native tiering cannot work this way: SecondaryTierManager is handed CPU pool slot addresses and never sees GPU memory, so blocks have to cascade through vLLM's pinned pool, which is a residency layer you size and which holds a second copy of every block.
• Restart and cross-instance reuse. Keys are content hashes namespaced by model, dtype, tensor-parallel size, GPU block size, and KV layout. A restarted engine hits blocks its predecessor stored, and so does a second instance with the same configuration and the same PYTHONHASHSEED.
• MemKV owns capacity. The plugin tracks existence only. There is no eviction bookkeeping to configure on the vLLM side.
• Config-only wiring. No vLLM patch and no MemKV server change.

Limits in this version

• One KV cache group. The worker asserts this at startup.
• One offload key per GPU block. Do not set block_size.
• The same tensor-parallel topology. Keys are rank-qualified because the stored bytes are one rank's shard. A cache written under TP=8 is readable only by TP=8.
• RDMA only. There is no TCP fallback on this path.

Other settings

A transfer that overruns memkv_wait_timeout_s is abandoned and reported as a failure. Its blocks are never marked present, so no later load can read a partly written value.

References

• vLLM KV offloading guide
• KV cache sizing — where the value_len terms come from
• RoCEv2 setup — lossless fabric for the RDMA path
• The other two vLLM paths: LMCache and native tiering