KV Cache Sizing

This page shows how to turn a benchmark cell such as c=128, 64K into bytes of KV cache, how those bytes divide across GPUs, and how to choose the MemKV slot size that holds them.

64K is tokens, not bytes

In inference benchmark tables, the context column is the prompt length in tokens. A row that reads 64K means 65,536 tokens of input. It has nothing to do with KiB, MiB, or GiB.

A token is a unit of text that the model's tokenizer produces. The Llama-3.1 tokenizer has 128,256 entries, and English averages about 0.75 tokens per word. So a 64K-token prompt is about 50,000 English words, or a 200-page book.

The number of bytes depends on the model. For Llama-3.1-70B at BF16, a 64K prompt needs 20 GiB of KV cache per session. The rest of this section shows the calculation.

What the KV cache is

For each token, a transformer computes key (K) and value (V) vectors in every layer and keeps them, so that later tokens can attend to earlier ones without computing them again. These stored vectors are the KV cache.

Every token, whether from the prompt or generated, adds one K and one V vector per layer per KV head. So the KV cache grows linearly with context length: each token adds a fixed number of bytes.

The formula

For one session (batch size 1) with a context of N tokens:

KV bytes per session = 2 × N × num_layers × num_kv_heads × head_dim × bytes_per_elem
                       │   │       │              │             │            └── 2 for BF16/FP16
                       │   │       │              │             └────────────── e.g., 128
                       │   │       │              └─────────────────────────── after GQA reduction
                       │   │       └──────────────────────────────────────── e.g., 80 for 70B
                       │   └─────────────────────────────────────────────── tokens of context
                       └─────────────────────────────────────────────────── K and V are stored separately

For one token of context:

KV bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_elem

LMCache's calculator publishes this per-token value for each model. For Llama-3.1-70B:

2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB per token (≈ 0.31 MiB)

Worked example: Llama-3.1-70B

The model's Hugging Face config.json gives:

Per-token KV (total across all GPUs)

2 × 80 × 8 × 128 × 2 = 327,680 bytes = 320 KiB

Per-session KV at common context lengths

Each value is the total KV cache for one session, summed across all the GPUs that hold part of it.

Tensor-parallel sharding

With tensor parallelism (TP) of N, the KV heads are split across N GPUs. The number of KV heads must divide evenly by TP (num_kv_heads % TP == 0), so for Llama-3.1-70B, with 8 KV heads, TP can be 1, 2, 4, or 8.

At TP=8, each GPU holds 1 KV head. The KV per token on each GPU is:

2 × num_layers × (num_kv_heads / TP) × head_dim × bytes_per_elem
2 × 80         × (8 / 8)             × 128      × 2
= 40,960 bytes = 40 KiB per token, per GPU

For one session at 64K context:

65,536 × 40 KiB = 2,621,440 KiB = 2.5 GiB per GPU

Across the 8 GPUs, that is 2.5 × 8 = 20 GiB total per session: the same total as before, split eight ways. TP divides the KV cache across GPUs. It does not change the total.

Adding concurrency

A load generator run with --concurrency C keeps C requests in flight. Each request has its own KV cache, so the total live KV cache is:

total live KV = C × KV_per_session

For example, at c=128 and 64K context:

128 × 20 GiB = 2,560 GiB = 2.5 TiB of live KV state across the cluster

An 8× H200 server has 8 × 141 GB ≈ 1.05 TiB of HBM, and about 0.93 TiB of that is left for KV after the 70B model weights. 2.5 TiB is far more than 0.93 TiB, so most sessions cannot stay in HBM. They must be evicted to an offload tier or computed again on the next request. This is where an external KV cache tier is needed.

The agentic sweep in inference performance measures this point. It raises concurrency step by step for each model. Below the point where live KV exceeds HBM, the tier costs throughput. Above it, the tier keeps the GPUs decoding instead of computing prefill again. Use the calculation on this page to find which side your deployment is on.

When MemKV helps

MemKV helps in one way: it saves prefill compute on the GPU. Each KV block read from MemKV replaces prefill that the GPU would otherwise run. The longer the prefill it replaces, and the more often a prefix repeats, the larger the gain.

MemKV needs prefix reuse. If almost every prompt is unique, MemKV has nothing to return: each request still runs full prefill on the GPU, and storing its KV cache only adds I/O.

MemKV helps with:

• Long prompts where prefill is the largest cost, such as RAG over large documents, codebases in the context, and transcripts.
• Prefixes reused across requests, such as shared system prompts, multi-turn conversations, agent loops, and batch evaluation over the same context.
• Traffic larger than the prefix cache in GPU HBM, so prefixes are evicted and would otherwise be computed again.

MemKV does not help with:

• Mostly unique prompts with no shared prefix.
• Short prompts, where prefill is already cheap.
• Working sets that fit in HBM and are never evicted. This case has a cost: MemKV still stores KV but never reads it back, and the measured cost is a median 15% of throughput, up to 30% on the largest models. Size the deployment before you enable the tier.

How KV blocks reach MemKV

The inference engine stores the KV cache in fixed-size blocks. Recent vLLM releases use 16-token blocks by default (set with --block-size). For Llama-3.1-70B at TP=8, one vLLM block on one GPU is:

16 tokens × 40 KiB/token = 640 KiB per block per GPU

The engine groups consecutive blocks into larger transfers, so each transfer to MemKV is several MB to tens of MB. At these sizes MemKV reads at high bandwidth.

The client sends up to max_batch_size keys to a server in one control message. The control messages use RC (Reliable Connection) queue pairs. The server then moves the KV data for each key over RDMA, using DC (Dynamically Connected) transport on mlx5 NICs and RC on other NICs. A DCI (DC initiator) is one sender on the DC transport. These settings control the batching and parallelism:

For 70B BF16 at TP=8, 64K context, and c=128, this gives about:

• Per request: 20 GiB in total, 2.5 GiB per GPU read from MemKV.
• In total: 2.5 TiB of live KV cache. The part that does not fit in HBM goes to MemKV.
• On the network: batches of up to max_batch_size keys, with several batches in flight on the connection pool.

None of this involves "64 kilobytes."

Choosing storage.block_size

storage.block_size is the size of one slot on the drive. MemKV gives every value slots of its own:

• A value smaller than a slot takes one slot. The rest of the slot stays empty, and no other value can use it.
• A value larger than a slot is split across as many slots as it needs. Only the last of them is partly empty.

So the drive is used well when a value fills its slot or spans several slots, and wasted when a value is much smaller than a slot. This applies to the direct and file storage modes.

How much of a slot a value can use depends on the mode:

• In direct mode, keys are stored outside the slots, so a value fits in one slot if it is at most block_size.
• In file mode, each slot also holds a 544-byte header (32 bytes plus room for the longest key), so a value fits in one slot if it is at most block_size − 544 bytes. A value of exactly 1 MiB therefore takes two 1 MiB slots in file mode.

What one value is

The value size comes from the engine, not from MemKV. For vLLM:

• With direct offload (MemKVOffloadingSpec), one value is one GPU KV block from one tensor-parallel worker: vLLM block size in tokens × KV bytes per token per GPU.
• With CPU tiering (TieringOffloadingSpec), one value is one offloaded block across all tensor-parallel ranks, sized by the block_size you give the connector.

The KV bytes per token come from the formula earlier on this page, and depend on the model, the KV data type and the tensor-parallel size.

Measure your value size

You do not need to work the size out. After vLLM has stored some KV, divide memkv_write_bytes_total by memkv_write_total from a MemKV server's /v1/metrics. The vLLM plugin writes one value per request, so this average is the value size. The counters cover every client of the server, so take the reading while vLLM is the only one writing to it.

If the value size is a small fraction of block_size, most of each slot is empty, and a smaller block_size would hold more KV on the same drive.

Examples

For the DeepSeek-V4-Flash case, the console of a server with 1 MiB slots showed 5,669 stored blocks using 5.5 GiB, exactly one slot each. With the default 4 MiB slots, the same drive holds a quarter as much KV.

Which size to use

• Keep the default 4 MiB for production unless you have measured your values. Large models, and offload blocks of many tokens, produce values of several megabytes, which 4 MiB slots hold with little waste. Fewer, larger slots also mean less metadata per byte stored.
• If your values are well under 4 MiB, use the smallest size that holds one value in one slot. For DeepSeek-V4-Flash with direct offload that is 1 MiB, which stores four times as much KV.
• In file mode, leave room for the 544-byte header. If your value is exactly a power of two, choose the next size up.

Changing the size

The slot size is fixed when MemKV formats a drive or creates a file. Changing it means starting with an empty cache:

• file mode. If the config's block_size differs from the file's, memkv start stops with a block size mismatch error. Stop MemKV, move the old file away or delete it, and start MemKV again. It creates a new file with the new slot size.
• direct mode. A formatted drive keeps the slot size it was formatted with, whatever the config says. To change it, reformat with memkv setup --drives <device...> --force --shard-size <KB>: --shard-size 1024 gives 1 MiB slots and --shard-size 4096 gives 4 MiB. This erases the drives.

Reference numbers for other Llama-family models

Per-token KV at BF16, total across all GPUs:

For any model, use the same formula with the values from its config.json. Check the result against LMCache's KV calculator before you quote it.

Summary

When you see a benchmark cell like c=128, 64K:

• 64K = 65,536 input tokens (not bytes, not anything else).
• c=128 = 128 simultaneous in-flight requests (--concurrency 128 on the load generator).
• The bytes follow from 2 × N × num_layers × num_kv_heads × head_dim × bytes_per_elem for one session, multiplied by c for the aggregate.
• For any specific model, run that arithmetic with the model's num_layers, num_kv_heads, head_dim, and dtype — or paste it into the LMCache calculator.

Sources

• LMCache — KV Cache Size Calculator — canonical per-model KV per-token reference
• VMware Cloud Foundation Blog — LLM Inference Sizing and Performance Guidance
• Spheron — GPU Memory Requirements for LLMs: VRAM Calculator
• AWS Neuron — Training Llama-3.1-70B with Tensor Parallelism — TP sharding constraints
• JAX Scaling Book — Serving LLaMA 3-70B on TPUs — KV-head sharding strategy
• Hugging Face — meta-llama/Llama-3.1-70B-Instruct config.json — the architectural source of truth