vLLM direct KV offloading + MemKV

MemKVOffloadingSpec makes a MemKV cluster the offload medium for vLLM's native KV offloading. The vLLM worker reads each KV block from the GPU, packs it into one value, and MemKV servers move it over RDMA. vLLM's CPU pool is not used, so there is no second copy and no pool to size. For a comparison with the other two vLLM paths, see Choose a vLLM path.

The recommended setup is short: install the plugin, point it at MemKV, and serve with the defaults. The default memkv_scratch_medium: auto chooses the faster staging location for your host. The tuning sections after that are for when you need them.

What you need

• A running MemKV cluster. See the Quick Start.

• A MemKV license file and the auth key (32 bytes, hex-encoded).

• The memkv-vllm package. It runs on Linux only.

• An RDMA NIC on the GPU host. This path has no TCP fallback. If the client cannot reach a NIC, the worker stops at startup.

• An NVIDIA or AMD GPU whose memory the NIC can register:

  • NVIDIA (CUDA 12+): as a dma-buf, or through nvidia-peermem. The client tries dma-buf first.
  • AMD (ROCm 6+, Instinct GPUs): through amdgpu's own peer-memory client, which the MLNX_OFED or DOCA RDMA stack accepts, or as a dma-buf, which needs rdma-core 34 or later and a NIC driver that registers dma-bufs. The client tries the peer-memory client first when it is registered.

  Each worker tries the registration at startup; see Choose where KV stages for what happens when it fails.

• A vLLM version whose offloading API can load a spec from a module path. Check before you serve:

  python -c "from vllm.v1.kv_offload.base import OffloadingSpec, CanonicalKVCaches; print('ok')"

  If the import fails, your vLLM is too old for this plugin. vLLM marks the offloading API as experimental, so check again when you change vLLM versions.

• PYTHONHASHSEED set to the same value, for example 0, on every instance that should share KV cache. vLLM 0.25 and earlier, and later versions with --prefix-caching-hash-algo xxhash, seed block hashes differently in each process; without a fixed seed, lookups from another instance, or from the same instance after a restart, miss. Later versions with the default sha256 do not need it. The plugin logs a warning at startup when the seed would vary.

• --enable-cumem-allocator, if the environment sets PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, as many container images and launch scripts do. Otherwise vLLM refuses to start OffloadingConnector. The flag moves the model weights and KV cache into vLLM's own memory pool, where expandable segments are off.

Step 1: install the plugin

pip install memkv-vllm   # or a local wheel: pip install memkv_vllm-*.whl

Step 2: point the client at MemKV

The plugin uses the standard MEMKV_* environment variables, like the other MemKV plugins:

export MEMKV_SERVERS=10.0.0.17:9900
export MEMKV_AUTH_KEY=<64-hex-char key>
export MEMKV_LICENSE=/etc/memkv/license      # JWT or path
export MEMKV_RDMA_DEVICES=mlx5_0,mlx5_1,...  # optional; see below

You can instead mount a client.yaml file and set MEMKV_CONFIG=/path/client.yaml.

Each tensor-parallel rank picks the NIC that shares a PCIe switch with its GPU. Where several NICs are equally close, as in a VM that hides the PCIe tree, the GPUs that tie on them take one each in PCI address order. If you leave MEMKV_RDMA_DEVICES unset, every NIC on the host is a candidate. Set it to limit the choice, for example to keep MemKV off a NIC used by another fabric; each rank then picks the closest NIC in your list. A list with one device puts every rank on that NIC, which you want only on a host with one NIC.

memkv net topo prints the GPU and NIC layout the ranks use.

Step 3: serve with the spec

PYTHONHASHSEED=0 vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "spec_name": "MemKVOffloadingSpec",
      "spec_module_path": "memkv_vllm.spec"
    }
  }'

Do not set block_size in kv_connector_extra_config. This version needs one offload key per GPU block, and the worker stops at startup if block_size is set.

Check the startup log

Each rank logs one line:

MemKV GPUDirect worker: rank=0 tensors=80 layers/block=80 value=0.62MiB
  threads=4 chunk=64 blocks scratch=160MiB HBM

• value= is the size of one block, across all layers, for one rank.
• chunk= is how many blocks share one staging buffer.
• scratch= is the total size of the staging buffers for this rank.

If the line shows chunk=1 blocks, see Size the staging pool.

Choose where KV stages

Each transfer passes through a small pool of staging buffers registered with the NIC. The memkv_scratch_medium setting in kv_connector_extra_config chooses where they are:

"memkv_scratch_medium": "auto"    // default; or "gpu", "host"

• host: the buffers are in pinned host memory. Each block crosses PCIe once between the GPU and the buffer. This works at full speed wherever the NIC is.
• gpu: the buffers are in GPU memory (HBM), and the NIC reads and writes them directly. This is faster only when every GPU on the host has a NIC on its own PCIe switch.
• auto (recommended): reads the PCIe layout and chooses gpu only when every GPU on the host has such a NIC and the NIC accepts a registration of GPU memory, and host otherwise.

nvidia-smi topo -m shows the layout (rocm-smi --showtopo on AMD): PIX means the GPU and NIC share a PCIe switch; NODE and SYS mean traffic crosses the CPU root complex, which is slow. The rule applies to the whole host: one GPU without a PIX NIC slows every rank that stages in HBM, including ranks that have one. On such a host, host is the right choice, and it costs almost nothing. The measurements show the difference: up to 3.8x on restores, in either direction.

Both settings store the same bytes under the same keys, so data written under one is readable under the other.

To override auto, set gpu or host. If you set gpu on a host that does not qualify, the plugin logs a warning that says why.

Each worker registers its GPU staging buffers with the NIC at startup, before it serves. If the NIC refuses them, auto stages in host memory and logs the cause, and an explicit gpu stops the worker with it. On AMD without amdgpu's peer-memory client, a refusal usually means the NIC driver or libibverbs cannot register dma-bufs, or that the kernel allows no PCIe peer-to-peer path between the GPU and the NIC, which amdgpu answers by moving the buffer into system memory. memkv net show reports whether libibverbs can register dma-bufs and which peer-memory clients are registered. The startup log names each rank's NIC, its PCIe class, and the decision for the host:

MemKV GPUDirect worker: rank=4 gpu=CUDA 12.9 nic=mlx5_4(PIX) tensors=80 ...
MemKV rank=4: staging in HBM (nic=mlx5_4 affinity=PIX, host has a NIC per GPU: True)

gpu= names the GPU runtime the worker's torch was built for (CUDA or ROCm). memkv net topo prints the same table and decision without starting vLLM.

memkv bench --auth-key <key> \
  --servers <rail0>:9900,<rail1>:9900 --devices mlx5_0,mlx5_1 \
  --vllm=host --kv-model llama-3.1-70b --tp 8 --gpu-block-size 64 \
  --concurrency 32

Size the staging pool

A block's layers are not next to each other in GPU memory, so the plugin gathers each block into one buffer and stores it as one value. Those buffers are the staging pool.

With the host medium, the pool uses a few hundred MiB of pinned host memory and needs no planning. With the gpu medium, the pool uses GPU memory that the KV cache would otherwise use, and you must leave room for it. With auto, plan for GPU memory on hosts where every GPU has a PIX NIC.

Settings

The block size (value_len) comes from the model:

value_len = num_layers × 2 × block_size × (num_kv_heads ÷ TP) × head_dim × bytes_per_elem

Here 2 is for K and V, block_size is the vLLM GPU block in tokens, num_kv_heads is after GQA reduction, and bytes_per_elem is 2 for BF16 and FP16. The pool then is:

chunk_blocks = clamp(floor(scratch_mb_per_thread × 1 MiB ÷ value_len), 1, 64)

scratch_total = num_threads × chunk_blocks × value_len

The budget is an upper limit per buffer, not the amount allocated. The pool rounds down to whole blocks and holds at most 64 blocks per buffer, so it often uses less than the budget. Plan GPU memory from scratch_total, or from the scratch= value in the startup log, not from num_threads × scratch_mb_per_thread.

• Small blocks reach the 64-block cap. Llama-3.1-70B reaches 64 blocks with 40 MiB of its 128 MiB budget, so a larger budget changes nothing.
• Large blocks reach the budget. A GLM-5.2 block is about 3.5 MiB, so the budget runs out at 36 blocks. Here the budget controls the pool: 48 MiB gives 13 blocks and 182 MiB.

At the defaults, the pool uses at most 512 MiB per GPU: 64 MiB for a small-block model, 160 MiB for Llama-3.1-70B, 504 MiB for GLM-5.2. Each rank has its own GPU and its own pool, so 512 MiB is the figure that competes with that GPU's KV cache. On an 8-GPU node, that is 4 GiB in total.

Leave room in gpu_memory_utilization

This applies only to ranks that stage in GPU memory.

vLLM sizes the KV cache to fill gpu_memory_utilization before the plugin allocates its pool, so the pool is not inside vLLM's budget. If no memory is left, the pool takes the margin you left for the driver, or fails to allocate and stops the engine at startup. Lower gpu_memory_utilization by the pool's share of GPU memory:

headroom = scratch_total ÷ total_gpu_memory

For 504 MiB of scratch on a 141 GB GPU, that is 0.004, so --gpu-memory-utilization 0.90 becomes 0.89. Round down: a spare 100 MiB is cheaper than a failed start. To avoid this calculation for each model, reserve the maximum once: 512 MiB at the defaults, 0.38% of a 141 GB GPU.

The KV cache shrinks by the amount you reserve:

tokens given up = scratch_total ÷ (value_len ÷ block_size)

For Llama-3.1-70B BF16 at TP8, one rank holds 40 KiB per token:

That is about one long request's worth of cache, in exchange for serving prefixes from MemKV across restarts and instances.

The gather and scatter steps also use short-lived buffers of about 1% of the pool: 0.5 MiB per thread and 2 MiB in total for the 160 MiB Llama-3.1-70B pool. The reservation above covers them.

Change the settings only for a reason

Start with the defaults.

• GPU memory is tight. Lower memkv_scratch_mb_per_thread. Smaller chunks need more RDMA round trips for the same blocks.
• Transfers do not keep up. Raise memkv_num_threads. Memory use grows in proportion, because each thread has its own buffer. Past the 64-block cap, more threads is the only way to put more data in flight.
• Keep chunk_blocks above 1. See the warning above.

More pool than the network can keep busy buys no throughput and only takes KV cache from the engine. To find the smallest pool that saturates your network, run memkv bench --vllm:

memkv bench --auth-key <key> \
  --servers <rail0-addr>:9900,<rail1-addr>:9900 \
  --devices mlx5_0,mlx5_1 \
  --vllm --kv-model llama-3.1-70b --tp 8 --gpu-block-size 16 \
  --gpu-devices 0,4 --concurrency 1,2,4

It tests concurrency levels against the plugin's staging budgets, reports KV tokens per second, and recommends the smallest pool that saturates the network, with its cost in KV cache tokens. Run it on your own hosts; results do not carry from one host to another.

Set up the network rails

• Give each rail a MemKV server address on that rail's own subnet. A rail that talks to an address on another rail's subnet stalls. The bench matches rails to servers by subnet, so --servers and --devices do not need to be in the same order.
• Each MemKV server must serve on every rail: list all its NICs in rdma.devices and bind the data port to the wildcard address. See rdma.devices. If a server uses one NIC, GPUs on the other rails cross the root complex, whatever the plugin setting.

Limits in this version

• Tensor parallelism only. Startup fails with pipeline parallelism or prefill context parallelism.
• One offload key per GPU block. Do not set block_size.
• The same tensor-parallel size. Each stored value is one rank's shard, so a cache written with TP=8 is readable only with TP=8.
• RDMA only. There is no TCP fallback.
• A failed transfer stops the engine on every vLLM release through 0.32.0. A store or load can fail because of a dead rail, a storage error, a block evicted after lookup, or a transfer that outlives memkv_wait_timeout_s. vLLM's connector then stops the EngineCore. A vLLM patch makes the connector compute the affected blocks again instead. No vLLM release includes it; contact MinIO support for it, and set kv_load_failure_policy to "recompute" in the KVTransferConfig with it. With the default "fail" policy, a request whose load failed ends with an error.

How blocks are keyed

Each block key starts with a prefix built from the model name, the KV cache dtype and layout, the tensor-parallel size, and memkv_spec_prefix. The workers add a fingerprint of the weights they load:

• A model from the Hugging Face cache: its snapshot commit.
• A local directory: a hash of config.json and of each shard's safetensors header and sampled tensor bytes.

So a new checkpoint served under the same model name starts a new namespace instead of restoring KV cache computed by the old one. If the startup log shows weights unknown, the weights could not be identified; set memkv_spec_prefix to the checkpoint's revision.

At startup, each worker publishes its prefix to MemKV and the scheduler uses it, so the two sides always agree. If a model changes a keyed setting while its workers load, the scheduler logs using the workers' prefix and serves normally.

Startup fails in two cases:

• worker ranks key KV blocks differently: ranks disagree, usually because nodes have different model files or different memkv-vllm versions. The message prints each rank's key description and weights fingerprint.
• published no key prefix: a node runs an older memkv-vllm, or a different model, tensor-parallel size, or memkv_spec_prefix.

At runtime, a warning that blocks this engine stored were missing when checked again means restores rarely succeed, usually because MemKV evicts blocks sooner than memkv_lookup_hit_ttl.

Model compatibility

Other settings

Set these in kv_connector_extra_config:

A transfer that runs longer than memkv_wait_timeout_s is abandoned and reported as a failure. Its blocks are never marked present, so no later load can read a partly written value.

Measurements

These results come from one test host. Fabric, drive latency, and how many ranks share a rail all change them, so do not carry them to another host; measure your own with memkv bench.

Host and GPU staging

Measured on 8x H200 against two MemKV nodes over 2x400 Gb, 32 concurrent jobs, restoring 2.5 MiB blocks with the returned bytes checked against what was stored:

Restore reached 87% of line rate through host memory and 23% through GPU memory on this host. On this host each GPU sits under its own PCIe switch, and each NIC shares a switch with exactly one GPU, so most GPUs have no PIX NIC.

What gpu costs without PCIe affinity

Measured on one 400 Gb rail restoring 2.5 MiB blocks at 32 concurrent jobs, changing only which GPUs the traffic came from:

In the last row the whole rail delivers a third of its line rate, not only less per GPU.

One GPU without a local NIC slows the whole host

Same host, both 400 Gb rails, Llama-3.1-70B at TP=8, restoring 2.5 MiB blocks at 32 concurrent jobs, changing only which GPUs staged in HBM:

What decides throughput is whether every GPU in the staging set has a PIX NIC, not how many do. One GPU without one slows every GPU in the set. It also raises tail latency: staging all eight GPUs in HBM gave a p99/mean of 2.05, while the two-PIX set and host stayed at 1.00–1.04. This is why auto judges the whole host, and why host is right on any host where one GPU lacks a PIX NIC. One NIC can be local to several GPUs on the same switch, so what matters is coverage, not the number of NICs.

Two other causes were tested and ruled out. Limiting the staging set to one CPU socket's GPUs still showed the drop, so CPU interconnect saturation is not the cause. Disabling PCIe ACS redirects (ReqRedir, CmpltRedir, UpstreamFwd) on every switch port moved 17.5 GB/s to 18.2, which is noise.

Routing ranks without a local NIC through a GPU that has one, over NVLink, was also measured. Compared with staging on each rank's own GPU, it gained +31% on store and +3.4% on restore, so it does not fix restores, and the plugin does not use it. For full throughput on every rank, give each GPU a NIC on its own PCIe switch, as rail-optimized nodes do. Otherwise, auto uses host, which on this machine restored 3.8x faster than gpu.

Pool size against network

With one rank per rail on a 2×400 Gb host against two MemKV nodes, restore reached 38.3 GB/s at 4 concurrent jobs of 64 blocks with a 160 MiB pool. The bandwidth-delay product at that rate was 134 MiB, so the default pool was about 1.2× the product: near the point where more pool stops helping. Throughput was still rising at the top of that test, which means more memkv_num_threads could help.

References

• vLLM KV offloading guide
• KV cache sizing: where the value_len terms come from
• RoCEv2 setup: lossless fabric for the RDMA path
• The other two vLLM paths: LMCache and native tiering