sglang + MemKV

sglang's HiCache is a tiered prefix cache — a device cache on the GPU plus a host cache in pinned CPU RAM — backed by a pluggable storage tier through StorageBackendFactory. The MemKV plugin slots in alongside the built-in file, nixl, mooncake, hf3fs, eic, aibrix, and simm backends. sglang never sees MemKV directly: it talks to the HiCache controller, the controller talks to the plugin, the plugin talks to a MemKV cluster.

The rest of this page is the wire-up: HiCache lands evicted host-cache pages in MemKV and reads them back on prefix-cache hits.

What you need

• A running MemKV cluster (one or more nodes).
• A MemKV license file (minio.license).
• The MemKV auth key (32-byte HMAC, hex-encoded).
• Network access to PyPI for the memkv-sglang package.
• An installation of sglang (the official lmsysorg/sglang image is the simplest path).
• One or more RDMA NICs visible on the GPU host if you want the fast path — set MEMKV_RDMA_DEVICES=mlx5_0,mlx5_1 to bind them.

Step 1: bring up MemKV

Use the standard MemKV deployment flow. Once the cluster is up, each node listens on TCP :9900 for the wire protocol and HTTP :9901 for admin (by default; data_port + 1).

Step 2: install the plugin

The memkv-sglang plugin is published on PyPI for x86_64 and aarch64 Linux with glibc 2.28 or newer. Install it into the same Python environment sglang runs in:

pip install memkv-sglang

Inside a container, pip-install at startup (see the docker run example in Step 4).

Step 3: configure the MemKV connection

The plugin reads the standard MemKV config chain — MEMKV_CONFIG yaml first, then MEMKV_* env vars. For most operators the env vars are enough:

export MEMKV_SERVERS="host-a:9900,host-b:9900"
export MEMKV_AUTH_KEY="<64-hex-char auth key>"
export MEMKV_TRANSPORT=auto          # or rdma; tcp only without RDMA
export MEMKV_LICENSE=/path/to/minio.license
export MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1"     # required for the RDMA fast path
export MEMKV_STAGING_SIZE_MB=1024             # per-rank staging pool
export MEMKV_STAGING_SLOT_MB=16               # per-op staging slot

Keep MEMKV_TRANSPORT=auto, or set rdma to fail loudly when no RDMA NIC is visible. Inside Docker, the RDMA fast path needs --device=/dev/infiniband, --cap-add=IPC_LOCK and --ulimit memlock=-1 on the run command, as in the example below. What a missing MEMKV_RDMA_DEVICES does depends on the mode: in auto the plugin has no NIC to bind and falls back to TCP, while in rdma it is a startup error and the process exits.

Set MEMKV_TRANSPORT=tcp only when RDMA is not reachable at all, for example a container without /dev/infiniband or a routed network without RoCE. With tcp, the startup log still shows the RDMA device initializing, but every KV transfer goes over TCP; check for transport: Auto or transport: Rdma in the Creating memkv engine line. HMAC authenticates every frame but does not encrypt it, so keep TCP on a network you control — see Encryption.

The staging knobs control how much pinned host memory each rank reserves for RDMA staging buffers. With TP=8, the per-host commitment is 8 × MEMKV_STAGING_SIZE_MB.

Step 4: launch sglang with HiCache pointed at MemKV

Tell sglang to enable HiCache and select the dynamic storage backend pointing at the plugin's class:

docker run -d --name sglang-memkv \
    --runtime=nvidia --net=host --shm-size=64g \
    --device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1 \
    -e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    -e MEMKV_SERVERS="host-a:9900,host-b:9900" \
    -e MEMKV_AUTH_KEY="$AUTH_KEY" \
    -e MEMKV_TRANSPORT=auto \
    -e MEMKV_RDMA_DEVICES="mlx5_0,mlx5_1" \
    -e MEMKV_LICENSE=/minio.license \
    -v /path/to/models:/inference-models:ro \
    -v /path/to/minio.license:/minio.license:ro \
    --entrypoint bash \
    lmsysorg/sglang:<tag> -lc '
      pip install memkv-sglang &&
      python -m sglang.launch_server \
        --model-path /inference-models/<model-dir> \
        --host 0.0.0.0 --port 8810 \
        --tp <N> \
        --trust-remote-code \
        --enable-hierarchical-cache \
        --hicache-storage-backend dynamic \
        --hicache-storage-backend-extra-config "{\"backend_name\":\"memkv\",\"module_path\":\"memkv_sglang.backend\",\"class_name\":\"MemKVHiCacheStorage\"}"
    '

A clean startup logs these lines for each tensor-parallel (TP) rank, in order:

• Successfully installed memkv-sglang-<version> — pip ran fine.
• Creating dynamic storage backend 'memkv' (memkv_sglang.backend.MemKVHiCacheStorage) — sglang loaded the plugin module.
• INFO Creating memkv engine config=PluginConfig {...} — the plugin constructed its Engine.
• INFO memkv-client license verified plan=... — the license is valid.
• INFO Server mapped to rail server="..." — each configured MemKV server is registered with the router.
• INFO MemKVHiCacheStorage ready (servers=[...], rdma=..., suffix=...) — the plugin is fully online.

The suffix carries the model path, a fingerprint of its weights and, once sglang registers its host pool, the pool's class, layout, page size and bytes per token. For non-MLA models it adds the TP rank/size; for MLA models the TP dimensions are dropped on purpose (MLA already produces TP-shape-independent keys). When pipeline parallelism is used (pp_size > 1), the PP coordinates are appended too. Net effect: multiple sglang processes pointed at the same MemKV cluster do not collide on keys.

How HiCache uses MemKV

• Writes (host → MemKV): when HiCache decides to back a populated host-cache page up to storage (controlled by the --hicache-write-policy flag), the plugin's batched store call fires. Each page becomes one MemKV put keyed by the page hash plus the suffix above.
• Reads (MemKV → host): when the radix tree finds a prefix node whose host pages have been evicted, HiCache asks the plugin to refill them. The plugin fetches bytes from MemKV and writes them into the host-cache buffer before prefill resumes.
• Existence checks: before triggering a multi-page restore, the controller asks the backend which pages are present. The plugin groups keys by primary server and issues one batched exists call per server.

To force traffic through the storage backend during testing, shrink the host cache below the working set (--hicache-size must remain larger than the device cache, per sglang's protocol requirement, but a tighter ratio makes eviction routine).

What this integration buys

• Capacity beyond per-replica RAM. Once the host cache fills, evicted pages flow to MemKV instead of being dropped. The aggregate prefix cache becomes (host RAM × replicas) + MemKV cluster.
• Cross-replica sharing. sglang processes pointed at the same MemKV cluster share one chunk pool when they serve the same weights with the same KV layout. Two replicas serving the same workload can read each other's evicted pages.
• Durability. On-drive shards survive sglang restarts. (See Operational notes for the cold-restart caveat.)
• HMAC auth. Every op is HMAC-authenticated with the cluster-wide shared key.

Operational notes

• No cold-restart auto-warming. HiCache builds its radix tree lazily from request traffic, so a fresh process never asks the backend whether prior keys exist for the same model and configuration — it just re-prefills. The bytes are still in MemKV; the engine doesn't ask. This is sglang behavior, not the plugin's. Workaround: warm the radix with synthetic traffic on startup, or wait for sglang to add a discover-on-startup flow.
• Per-rank connections. With TP=N you will see N TCP sessions per MemKV server. The plugin does not share the client across ranks because each rank lives in its own scheduler subprocess.
• License is mandatory. The plugin verifies the license at construction; without one it raises at load time and sglang aborts startup. Mount the license file and set MEMKV_LICENSE (or configure via MEMKV_CONFIG).
• Keys follow the weights and the host pool's geometry. Each key carries a fingerprint of the weights at the model path (the hub snapshot commit, or a hash of a local checkpoint) and the pool's layout, page size and bytes per token. A new checkpoint, or a relaunch with another KV dtype, page size or layout, therefore does not read pages written under the old one. Two things it cannot see: a dtype change that keeps the bytes per token (fp8 e4m3 to e5m2), and a hub model served with --revision, which is fingerprinted at the cached default revision. After either, serve from a different model path or clear MemKV.
• Hybrid models (Mamba + MLA). sglang's HiCache + Mamba + MLA combination is bleeding-edge in upstream and trips its own internal kernels today. Pure-attention models (Llama, Qwen2.5, Mistral, Gemma) work cleanly. Mamba support in HiCache is on sglang's side to land.

Roadmap

• Cross-restart cold start. Either a sglang-side discover-on-startup pass, or an out-of-band warming script that walks expected prefix keys and asks the plugin to load them.
• RDMA fast path inside Docker by exposing /dev/infiniband and the matching capabilities; the plugin flips onto an RDMA READ direct-into-buffer path for reads.
• Pre-registered host pool MR so RDMA reads do not pay a per-call buffer registration cost.

References

• Upstream sglang project