Inference Performance

What this page measures

This page measures what MemKV gives an inference server: time-to-first-token (TTFT) and served throughput, with MemKV attached to vLLM as a KV offload tier.

The benchmarks page measures the wire instead: how fast bytes move between a client and MemKV, including under-load transport from this same campaign.

How we measure

Most published KV cache numbers use a fixed-length context, a guaranteed cache hit, and cold prefill as the comparison. That setup measures the cache's ceiling, not its value in production.

The numbers on this page use NVIDIA AIPerf's agentic-code workload instead: 1,000 synthetic coding-agent sessions with real think time (70% rapid agent steps ~2.5 s, 30% human pauses ~40 s) and contexts that grow to 124k tokens. The sessions are replayed at concurrency 8 → 256 with the engine's own prefix cache on, so MemKV gets credit only for reuse that the GPU cannot hold itself.

Campaign configuration

Every window in the sweep ran with these settings. Per-cell counts, intervals, and raw windows are in the campaign dataset, which is not published with these docs.

An "arm" is one engine setup in the comparison: the baseline without MemKV, or the engine with MemKV. The results table marks the rows whose confidence intervals (CIs) do not overlap the baseline's. We report P90 rather than P99: at 50–408 turns per cell, a P99 depends on the top one to five observations, so it shows the maximum rather than the tail.

Results by model

Verified August 2026. TP is tensor parallelism; C is concurrency.

The work-parity control

A cache that slows the engine receives less load, and less load shortens latency tails by itself. So every latency ratio above is reported next to the number of turns the same window completed. The chart below shows those turn counts: the pale bar is the work the baseline also did, and the solid bar is the extra work MemKV added.

Interactivity — throughput per GPU against per-user speed

A third way to read the sweep, after TTFT and completed turns, plots served tokens/s/GPU against per-user speed. Per-user speed is the median decode rate one request sees: one divided by its inter-token latency (ITL). Each arm is swept across closed-loop concurrency, so load rises from right to left, and up and to the right is better. SemiAnalysis InferenceX made this view popular.

These series come from a companion study on Nemotron-3 Ultra (550B/A55B FP8), not from the eight-model sweep above. It ran on the same 8× H200 and dual-rail MemKV platform, but at human pacing: every delay between turns is 30–60 s, slower than the agentic trace. It compares three arms: stock vLLM, LMCache MP with a DRAM L1 only, and native offload backed by MemKV.

The baseline peaks at 6,281 tok/s/GPU at C=64, then falls back to 4,092 as recompute takes over. The MemKV arm keeps climbing to 7,673 at C=128: 45% above the baseline at that rung, and 22% above the best point the baseline reaches anywhere on the sweep. The DRAM-only arm also gets past the cliff, but it peaks at 6,404 and falls off after C=128. That is the point where MemKV's NVMe tier starts returning what DRAM alone can no longer hold.

Compare the arms at the same interactivity. Near 17 tokens/s/user, both cache arms serve 28–31% more per GPU than the baseline. Near 13, the DRAM-only arm adds 21% and MemKV adds 45%. Neither tier is free before eviction starts: at C=32 and C=64 both lose 7–18% of baseline throughput. At C=16 the load is light enough that neither loses any.

The capacity cliff — both halves

The capacity cliff is the point where a model's live KV working set no longer fits in GPU memory (HBM). Results change in a step at the cliff, and the chart at the top of this page shows both sides. Both are measured the same way for every model: the cost is the mean over the concurrency rungs where the tier logged zero reads, and the gain is the best rung where the tier served more than 50 GiB.

Below the cliff, a write-through tier costs throughput: a median of −15% across the eight models, and up to −30%. Every block is stored and none is read back, because the GPU still holds everything the workload asks for.

Above the cliff, the baseline spends its time re-running prefill for evicted contexts, while MemKV keeps the GPUs decoding. On Llama 3.1 8B, the baseline without a tier falls back to 11,345 tok/s/GPU at C=256, while the MemKV arm climbs to 27,134 tok/s/GPU (+139%) and completes 2.59× the turns.

These figures include the cost of filling the tier, which ceiling measurements leave out. Stores run write-through inside every measured window, so there is no excluded warm-up pass. The only trim is the first 180 s of each window, applied the same way to every arm.

The fixed-context ceiling

The other way to measure (one request at a time, a fixed-length context, and a guaranteed hit compared with cold prefill) answers a different question: how fast the tier serves KV data when nothing else competes for it. Measured that way, MemKV reaches 20.5× P99 TTFT on H100 (24k contexts, output-validated) and 15.9–55.7× on AMD MI300X at long contexts.

Those results hold under the stated conditions, but they describe the ceiling, not behavior under load: every request hits, nothing queues, and there is no think time. Read them together with the agentic results above, not instead of them.

Where MemKV does not help

• Workloads with almost no prefix reuse. There is nothing to serve back.
• Models whose working set never outgrows HBM at your concurrency. They pay the write-through cost for no gain. Size the deployment first.
• Sliding-window-attention models. By design, they cache poorly. On gpt-oss-120b, half the KV is limited by the window, tier reads are about 10× lower than on dense models, and the results are statistically inconclusive. We report them as inconclusive.

MemKV helps most with prefix-reuse traffic past the capacity cliff: long shared system prompts, RAG over large documents, multi-turn agents, and batch evaluation over shared context.