MinIO MemKV Documentation

sglang's HiCache is a tiered prefix cache: a device cache on the GPU, a host cache in pinned CPU memory, and a storage tier that plugs in through StorageBackendFactory. The MemKV plugin is one such storage tier, alongside built-in backends such as file, nixl, and mooncake. sglang talks to the HiCache controller, the controller talks to the plugin, and the plugin talks to a MemKV cluster.

This page connects sglang to MemKV, so HiCache stores evicted host cache pages in MemKV and reads them back on prefix cache hits.

What you need

A running MemKV cluster. See the Quick Start.
A MemKV license file (minio.license).
The MemKV auth key (32 bytes, hex-encoded).
Network access to PyPI for the memkv-sglang package.
sglang. The official lmsysorg/sglang image is the simplest way to get it.
For the RDMA path, one or more RDMA NICs on the GPU host.

Step 1: install the plugin

The memkv-sglang plugin is on PyPI for x86_64 and aarch64 Linux with glibc 2.28 or later. Install it in the Python environment that sglang runs in:

pip install memkv-sglang

In a container, install it when the container starts, as in the docker run example in Step 3.

Step 2: configure the MemKV connection

The plugin reads the standard MemKV configuration: the MEMKV_CONFIG YAML file first, then MEMKV_* environment variables. Environment variables are usually enough:

export MEMKV_SERVERS="host-a:9900,host-b:9900"
export MEMKV_AUTH_KEY="<64-hex-char auth key>"
export MEMKV_TRANSPORT=auto          # or rdma; tcp only without RDMA
export MEMKV_LICENSE=/path/to/minio.license
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"     # required for the RDMA fast path
export MEMKV_STAGING_SIZE_MB=1024             # per-rank staging pool
export MEMKV_STAGING_SLOT_MB=16               # per-op staging slot

Keep MEMKV_TRANSPORT=auto. The transport choices are the same as for vLLM. In Docker, RDMA needs --device=/dev/infiniband, --cap-add=IPC_LOCK, and --ulimit memlock=-1, as in the example below.

The staging settings set how much pinned host memory each tensor-parallel (TP) rank reserves for RDMA transfers. With TP=8, each host reserves 8 × MEMKV_STAGING_SIZE_MB.

Step 3: launch sglang with HiCache pointed at MemKV

Enable HiCache and select the dynamic storage backend with the plugin's class:

docker run -d --name sglang-memkv \
    --runtime=nvidia --net=host --shm-size=64g \
    --device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1 \
    -e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    -e MEMKV_SERVERS="host-a:9900,host-b:9900" \
    -e MEMKV_AUTH_KEY="$AUTH_KEY" \
    -e MEMKV_TRANSPORT=auto \
    -e MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1" \
    -e MEMKV_LICENSE=/minio.license \
    -v /path/to/models:/inference-models:ro \
    -v /path/to/minio.license:/minio.license:ro \
    --entrypoint bash \
    lmsysorg/sglang:<tag> -lc '
      pip install memkv-sglang &&
      python -m sglang.launch_server \
        --model-path /inference-models/<model-dir> \
        --host 0.0.0.0 --port 8810 \
        --tp <N> \
        --trust-remote-code \
        --enable-hierarchical-cache \
        --hicache-storage-backend dynamic \
        --hicache-storage-backend-extra-config "{\"backend_name\":\"memkv\",\"module_path\":\"memkv_sglang.backend\",\"class_name\":\"MemKVHiCacheStorage\"}"
    '

Check the startup log

A clean start logs these lines for each TP rank, in this order:

Successfully installed memkv-sglang-<version>: pip installed the plugin.
Creating dynamic storage backend 'memkv' (memkv_sglang.backend.MemKVHiCacheStorage): sglang loaded the plugin.
INFO Creating memkv engine config=PluginConfig {...}: the plugin started its client.
INFO memkv-client license verified plan=...: the license is valid.
INFO Server mapped to rail server="...": the client registered a MemKV server. There is one line for each server.
INFO MemKVHiCacheStorage ready (servers=[...], rdma=..., suffix=...): the plugin is ready.

The suffix identifies the model and its KV cache layout, so several sglang processes that share a MemKV cluster do not overwrite each other's pages. See How pages are keyed.

How HiCache uses MemKV

Writes. When HiCache backs up a host cache page to storage, the plugin stores it in MemKV. The --hicache-write-policy flag controls when this happens.
Reads. When sglang finds a cached prefix whose host pages were evicted, the plugin reads them from MemKV into the host cache before prefill continues.
Existence checks. Before a restore, HiCache asks which pages MemKV holds. The plugin sends one batched check to each server.

To send more traffic through MemKV during testing, make the host cache smaller than the working set with --hicache-size. sglang requires the host cache to stay larger than the device cache.

What this integration gives you

Capacity beyond host memory. When the host cache is full, evicted pages go to MemKV instead of being dropped. The prefix cache becomes (host RAM × replicas) plus the MemKV cluster.
Sharing across replicas. sglang processes that serve the same weights with the same KV layout and point at the same MemKV cluster read each other's pages.
Durability. Stored pages survive sglang restarts, but sglang does not reuse them after a restart. See Operational notes.
Authentication. Every operation is authenticated with HMAC and the cluster's shared key.

How pages are keyed

Each key includes a fingerprint of the model weights (the Hugging Face snapshot commit, or a hash of a local checkpoint) and the host cache layout, page size, and bytes per token. For models other than MLA models, it also includes the TP rank and size; MLA keys do not depend on TP. With pipeline parallelism (pp_size > 1), it also includes the pipeline stage.

So a new checkpoint, or a restart with another KV dtype, page size, or layout, does not read pages stored by the old one. Two changes are not detected:

A dtype change that keeps the same bytes per token, such as fp8 e4m3 to e5m2.
A Hugging Face model served with --revision. Its fingerprint uses the cached default revision.

After either change, serve the model from a different path or clear MemKV.

Operational notes

A restarted process does not reuse pages stored before the restart. sglang builds its prefix tree from incoming requests and does not ask MemKV which pages it already holds, so a new process runs prefill again. This is sglang behavior; the pages are still in MemKV. To warm the prefix tree, send representative requests after startup.
Connections per rank. Each TP rank runs in its own process with its own MemKV client. With TP=N, each MemKV server sees N connections.
The license is required. The plugin checks the license when it loads. Without a valid license, sglang stops at startup. Mount the license file and set MEMKV_LICENSE, or set it in the MEMKV_CONFIG file.
Hybrid models. HiCache with Mamba and MLA together fails inside sglang today. Attention-only models such as Llama, Qwen2.5, Mistral, and Gemma work.

References

Upstream sglang project

sglang + MemKV