Configuration

Minimal config

A handful of fields vary between deployments; everything else has a sane default. Only override when you have a reason (covered below).

network:
  address: 0.0.0.0:9900 # data plane; admin auto-binds at port + 1

memory:
  block_size: 2 MiB
  max_size: 247 GiB # RDMA-pinned RAM (size to your box)

rdma:
  device: mlx5_0

storage:
  mode: direct # direct | file | memory
  block_size: 4 MiB
  drives:
    - { media: /dev/nvme0n1 }
    - { media: /dev/nvme1n1 }
    - { media: /dev/nvme2n1 }

That's the whole working config for a typical bare-metal node. The sections below explain each field, when the default is right, and what to think about before changing it.

Network

network:
  address: 0.0.0.0:9900
  # tls_cert: /etc/memkv/tls/fullchain.pem
  # tls_key:  /etc/memkv/tls/privkey.pem
  # auth_key: <64 hex chars, 32 bytes>
  throttle:
    enabled: false
    bits_per_sec: 200 Gbps
    burst_window_ms: 100

address — The single listen address for the server. The TCP listener carries the RDMA bootstrap (Connect) and, for clients whose RDMA data path is not active, the inline-bulk codes (TcpPut / TcpGet / TcpDelete + Exists); clients on the RDMA path move steady-state control onto RC SEND/RECV after bootstrap. The admin HTTP(S) endpoint binds at the same IP, port + 1 (so 0.0.0.0:9901 when address is 0.0.0.0:9900). Change the bind IP to scope MemKV to a specific NIC on multi-homed hosts; change the port if 9900/9901 conflicts.

tls_cert / tls_key — Set both to enable HTTPS on the admin endpoint. Use whenever the admin port is reachable beyond a trust boundary (cross-zone scrapes, ingress controllers). Setting only one is rejected at startup.

auth_key — HMAC-SHA256 shared key, 64 hex chars (32 bytes), used to authenticate every wire message between client and server. Required — the server refuses to start without one (loopback-only dev hosts can omit the key; the server derives a deterministic dev key from $HOME instead and logs a warning). Generate with openssl rand -hex 32. Can also be supplied via the MEMKV_AUTH_KEY env var (which overrides the yaml field if both are set); the Helm chart always sources it from a Kubernetes Secret. Clients must use the same key. Drift window between peers is ±60 s, so keep clocks in sync (chrony / ntpd).

Throttle

Caps the rate of bulk data transfers on this server. Off by default, in which case MemKV uses whatever line rate the fabric gives it. Turn it on when MemKV shares a rail with training or storage traffic and those workloads need reserved headroom.

enabled — Master switch. When false, the other fields are ignored and the data path runs no limiter code at all. Enabling it without bits_per_sec is rejected at startup.

bits_per_sec — The sustained ceiling. Accepts bps, Kbps, Mbps, Gbps, Tbps, or a bare number of bits per second. Prefixes are decimal, matching NIC and switch counters: 200 Gbps is 200,000,000,000 bits/s, about half of one 400GbE rail.

Unit matching folds case, and Gb, Gbit, and Gbps are all the same gigabit-per-second unit — so 200 GB is read as 200 gigabits per second, not gigabytes. Binary byte units (GiB, MiB) have no bit meaning and are rejected at startup rather than silently reinterpreted. If you are converting from a byte figure, multiply by 8 yourself: 25 GB/s of payload is 200 Gbps.

Rates above 1000 Tbps are rejected, whether written with a unit or as a bare number of bits per second. That is roughly 2500 times a single 400GbE rail, so a value that large is a units mistake rather than a fabric.

One cap covers both transports. MemKV's server issues every bulk transfer itself — the RDMA READ that pulls a client's write, and the RDMA WRITE that pushes a read — so RDMA and TCP draw on a single budget and a client straddling both cannot double its share. This holds for the batch paths (BatchRead / BatchWrite, TcpBatchRead / TcpBatchWrite) that the NIXL, LMCache, and SGLang integrations drive, because those fan out through the same two transfer calls.

Control messages (Connect, Lookup, Exists, Commit, Delete) are never metered against the cap, and the admin endpoint at port + 1 is a separate listener the cap never touches — health, status, and metrics stay available however saturated the data plane is.

One caveat for RDMA-attached clients. After bootstrap they carry steady-state control on the same RC path, and the RDMA worker pool (rdma.num_worker_threads, default 64) serves those control messages and bulk transfers from one set of threads. A fully saturated cap can leave every worker waiting on rate, and a control message then queues behind them: it is never charged against the cap, but it can be delayed. If you throttle aggressively and see control latency suffer, raise rdma.num_worker_threads. Clients on the TCP path are unaffected — that listener dispatches on its own pool.

burst_window_ms — The burst allowance: how far ahead of real time the limiter may run. Default 100. It is available from the first transfer onward, not only after an idle stretch, so a bursty inference workload is not paced into a sawtooth.

This makes the cap a long-run average, not a per-interval ceiling. Any interval can exceed bits_per_sec by up to the allowance — at the default that is 100 ms of extra rate, so a one-second window can carry about 110% of the cap; at burst_window_ms: 2000 the same window can carry three times it. The rate converges on bits_per_sec as the measurement window grows past the burst. Size the window against the shortest interval you need the cap to hold over, and set 0 if you need it to hold instantaneously.

Pick it from what you are protecting:

The window is honoured exactly at every rate, and it is the whole allowance — the limiter never runs further than this ahead of real time, no matter how long it has been idle. A transfer whose cost exceeds the allowance is paced on arrival rather than admitted and charged afterwards, so the ceiling holds even for the first transfer after a quiet stretch. Values above 10,000 are rejected, on the assumption that seconds were typed as milliseconds.

Note the interaction with a low cap: at 1 Mbps the default 100 ms window covers about 12 KB, so a 4 MiB block costs far more than the allowance and waits roughly 33 s. That is the cap doing what you asked, not a defect — but it means a rate low enough to matter should be set with storage.block_size in mind.

On the RDMA paths and on TCP reads the cap gates the transfer before any byte moves, so it bounds the instantaneous rate. On TCP writes the payload has already arrived by the time the server meters it — the limiter holds the response back, and backpressure reaches the client through its in-flight window. The average still converges on the cap, but a short inbound overshoot past the burst window is possible, bounded by how many requests the clients keep in flight. Treat the cap as a sustained-rate control, not a hard instantaneous ceiling on ingress.

The cap is per server, not per deployment. Ten servers at 200 Gbps admit 2 Tbps in aggregate; divide your fabric budget by the server count. MemKV does not coordinate a cluster-wide limit.

To confirm the cap is binding, query the rate of memkv_throttle_delay_seconds_total, not its value — it is a cumulative counter, so it stays permanently above zero once anything has ever been paced:

rate(memkv_throttle_delay_seconds_total[1m])

A non-zero rate while bulk traffic is flowing means transfers are waiting on the cap. A zero rate means the cap is not the bottleneck, which is what separates a throttled server from a slow drive. See Monitoring for the full metric set.

Memory

memory:
  block_size: 2 MiB
  max_size: 247 GiB

Sizes accept human units — B, KiB, MB, MiB, GB, GiB, TB, TiB, with optional whitespace.

max_size — The single most important tuning knob. This is RDMA-pinned host memory used as the staging area for all client I/O. Size it to ~75–85% of free RAM on the box, after subtracting OS, page cache, and other co-resident services. The default of 1 GiB is for development only — production deployments need hundreds of gigabytes.

block_size — Allocator slab inside the pool. Default 2 MiB matches the typical RDMA WR size and rarely needs changing. Raise only if you measure internal fragmentation hurting cache locality on very large objects. The implied block-count cap is max_size / block_size.

RDMA

MemKV runs on two RDMA fabrics: RoCEv2 on Ethernet and native InfiniBand. It reads the port's link layer at startup and addresses peers the way that fabric requires — by GID on RoCEv2, by LID on InfiniBand. Several settings below apply to one fabric only; each says which. See the RoCEv2 and InfiniBand runbooks for the fabric-side setup.

rdma:
  device: mlx5_0
  port: 1
  gid_index: 0 # 0 = auto-detect the routable RoCEv2 GID
  pkey_index: 0 # InfiniBand partition; 0 = the default partition
  traffic_class: 0
  service_level: 0
  mtu: 4096
  cq_depth: 4096
  timeout: 14
  retry_count: 7
  num_dcis: 256
  dc_key: 0
  num_worker_threads: 64
  num_responder_threads: 16

These defaults are tuned for a typical LAN-attached RoCE/IB cluster. Most operators only set device.

device — RDMA NIC to bind to. mlx5_0 is the first Mellanox/NVIDIA HCA. Set this when you have multiple HCAs and want to pin MemKV to a specific one (e.g., dedicating mlx5_0 to storage and mlx5_1 to a separate fabric). memkv setup auto-detects.

port — Physical port on the HCA. Leave at 1 unless you have a dual-port card (e.g., ConnectX-7) and want MemKV on the second port. Port enumeration is per-HCA, not per-host.

gid_index — RoCEv2 only. Selects which GID entry the QPs use. 0 auto-detects the routable RoCEv2 GID (the IPv4-mapped entry) from the NIC's GID table. This matters for lossless RoCE: GID index 0 on mlx5 is the RoCEv1 (Ethernet) entry, which has no IP header and therefore no DSCP, so a DSCP-trust fabric leaves it on the best-effort queue regardless of traffic_class. Auto-detect avoids that trap. Set a non-zero value to pin a specific GID entry (mixed RoCEv1/v2 environments, or VLAN-tagged subnets); if no RoCEv2 GID is found, it falls back to index 0 with a warning. InfiniBand addresses peers by LID and ignores this.

pkey_index — InfiniBand only. Index into the port's P_Key table, naming the partition every queue pair joins. 0 is the default partition: correct on RoCEv2 and on an unpartitioned InfiniBand fabric. Set it when the fabric is partitioned and this node belongs to a non-default partition — two ports in different partitions cannot exchange data, and the far end discards the packets with no error on the sender. MemKV refuses to bring RDMA up when the entry at this index is empty, naming the index — the server falls back to its non-RDMA pool and clients drop the rail, so check the startup log. Also settable via MEMKV_RDMA_PKEY_INDEX. See the InfiniBand runbook.

traffic_class — RoCEv2 only. Traffic class (the IP DSCP/ToS byte stamped on egress). 0 auto-detects it from the NIC's DCB config: MemKV reads the lossless PFC priority and the DSCP mapped to it, so RDMA lands in the same lossless+ECN queue your fabric configured. Set a non-zero value (the ToS byte, i.e. DSCP << 2 — e.g. 104 for DSCP 26) to pin it explicitly and skip detection. If detection finds nothing the value stays 0 (best-effort) and a warning is logged. InfiniBand carries no global routing header to hold a DSCP, so MemKV reports a configured value as ignored there. Also settable via MEMKV_RDMA_TRAFFIC_CLASS.

service_level — Applies to both fabrics, but means different things on each. On RoCEv2 it is the AH SL (0–7) stamped on egress alongside traffic_class: on mlx5 the egress scheduling class (which PFC queue the traffic uses, hence the lossless behavior) follows the SL, not the DSCP, so the SL must match the lossless PFC priority or RDMA stays on the best-effort queue even with the right DSCP. 0 takes the priority auto-detected with traffic_class. On InfiniBand the same value selects a virtual lane through the subnet manager's SL-to-VL map, which is unrelated to any Ethernet priority — do not carry a RoCE-tuned value across without meaning it in InfiniBand terms. Also settable via MEMKV_RDMA_SERVICE_LEVEL.

mtu — RDMA path MTU. 4096 requires jumbo frames end-to-end (HCA + switch + peer). Drop to 1024 or 2048 only if your fabric can't carry 4K MTU — performance scales with MTU, so size up wherever the network allows. The value must not exceed the port's negotiated active MTU: a path MTU larger than the link drops every RDMA data packet, and MemKV refuses such a configuration at startup rather than stalling the data path. Set the same value on the client (MEMKV_RDMA_MTU) and the server, and configure the fabric to match — see the RoCEv2 setup runbook.

cq_depth — Completion queue size. Sized for num_dcis × inflight margin. Raise if you see "CQ overflow" in logs under burst load; otherwise leave alone.

timeout — Local ACK timeout per QP, expressed as 4.096µs × 2^timeout. 14 ≈ 67 ms — generous for LAN where lost packets are rare. Lower (e.g., 12 ≈ 16 ms) on ultra-low-latency fabrics for faster failure detection; raise on lossy or oversubscribed networks.

retry_count — Number of retransmissions before a connection is declared dead. 7 is the RDMA spec maximum. Drop to 3–4 if you want faster client failover; the default favors resilience over snappiness.

num_dcis — Size of the DC Initiator pool (mlx5 only). DCIs are acquired round-robin per client request. 256 saturates most NICs. Increase if you see DCI contention under very high concurrency.

dc_key — 24-bit shared key between server and client for DC fabric-level auth. 0 means no key check (open). Set to a deployment-specific value when you want fabric-level isolation between MemKV clusters sharing the same RoCE network.

num_worker_threads — Parallelism for processing client writes/reads after RDMA arrival. Scale with available cores; if these threads peg CPU, raise — if they idle while QPs back up, the bottleneck is elsewhere.

num_responder_threads — Parallelism for posting response SENDs back to clients. Usually the smaller pool; raise only if SEND posting becomes a hot path.

Storage

storage:
  mode: direct # direct | file | memory
  block_size: 4 MiB
  flap_threshold: 5
  drives:
    - media: /dev/nvme0n1
      max_size: 2 TiB # cap usage to a subset of the physical drive
    - media: /dev/nvme1n1 # max_size omitted → use the full device
  # max_size: 16 GiB           # only used when mode: memory (no drives)
  eviction:
    enabled: true
    high_watermark: 70
    low_watermark: 60

mode —

• direct (default): one block lives on one drive. Raw NVMe via O_DIRECT + io_uring — bytes go straight to the drive. The fastest path. Linux only.
• memory: RAM-only backend, no persistence. For development, benchmarks, and ephemeral caches. Capacity comes from the top-level storage.max_size.
• file: mmap-backed regular files. No O_DIRECT, no NVMe ioctls — runs on any filesystem that supports mmap. Intended for single-node dev, CI, and the GPU-host co-located case where a spare NVMe isn't available. Bandwidth is bounded by the host filesystem; use direct for production. Each drives entry becomes one backing file, preallocated at max_size (default 4 GiB).

block_size — Object granularity on drive. 4 MiB balances metadata overhead (smaller = more index entries) against internal fragmentation (larger = more wasted space on sub-block writes). Drop to 1 MiB for many-small-objects workloads, raise to 8–16 MiB for streaming or large-object workloads.

flap_threshold — Failures tolerated before a drive is blacklisted and removed from the rotation. 5 is a reasonable middle ground. Lower (1–2) for paranoid fast-fail in production; raise for known-flaky drives you want to keep in service while you swap them. Ignored in file and memory modes.

drives[].media — Path to the NVMe device (mode: direct) or backing file (mode: file). For direct, raw block devices (/dev/nvmeXn1) — drives are formatted on first use, don't include OS drives. For file, regular file paths — files are created on first open at max_size.

drives[].max_size — Cap on capacity used by this drive. Omit to use the full physical capacity for direct, or the 4 GiB default for file. Useful when you want MemKV to use only a subset of a large drive.

storage.max_size — Top-level capacity for mode: memory. Sizes the in-RAM keyspace. Ignored for direct and file.

eviction — MemKV is a cache, so it reclaims its own capacity rather than filling up and refusing writes. Leave this on unless you have a reason not to.

Reclaim starts once storage reaches high_watermark percent full and releases the least recently used blocks until it is back under low_watermark. Freed space is TRIMmed before it is reused.

• enabled — true by default. Turning it off means the drives fill and then reject new keys until something deletes. Only sensible when the client manages retention itself and you would rather see writes fail than lose a cached block.
• high_watermark — 70. Where reclaim kicks in. Leaving headroom lets TRIM stay ahead of allocation instead of competing with it. Must be below 95, where the write path stops admitting new keys.
• low_watermark — 60. Where reclaim stops. Must be below high_watermark; the server refuses to start otherwise. The gap to high_watermark sets how much one pass does — narrow it and the evictor runs almost continuously; widen it and each pass drops more of the cache at once.

Which blocks go first depends on what MemKV knows about them. Blocks it has served or written since startup are reclaimed least-recently-used. After a restart on large drives the index is rebuilt lazily, so blocks not yet touched have no recency to sort by and are reclaimed oldest-first until they are.

Logging

logging:
  level: info # trace | debug | info | warn | error
  format: json # json | text

level — info is right for production. Drop to warn or error if log volume is a concern; raise to debug or trace only when debugging — trace is very chatty.

format — json for log shippers (Loki, ELK, Datadog). text for tailing locally during development.

Client configuration (MEMKV_CONFIG)

The NIXL plugin (libplugin_MEMKV.so) and the LD_PRELOAD shim (libmemkv_preload.so) load their settings from memkv-client. The canonical layout is a yaml file referenced by the MEMKV_CONFIG env var, with MEMKV_* env vars layered on top as overrides.

Resolution order, lowest to highest priority:

1. Built-in defaults (one local server, one rail, 256 MB cache, 8 connections)
2. MEMKV_CONFIG yaml file, if the env var is set
3. MEMKV_* env vars

# /etc/memkv/client.yaml — pointed at by `export MEMKV_CONFIG=...`
servers:
  - 10.0.0.1:9900
  - 10.0.0.2:9900
rdma_devices:
  - mlx5_0
  - mlx5_1
bind_addresses: # optional, one per server slot; ~ = bind any
  - 192.168.1.10
  - 192.168.1.11
cache_size_mb: 256
connect_timeout_ms: 5000
gid_index: 0
pkey_index: 0 # InfiniBand partition; 0 = the default partition
num_connections: 8
license: /etc/memkv/minio.license
auth_key: <64 hex chars, 32 bytes>
# transport: auto       # rdma | rdma-rc | tcp; default auto

The client uses the configured server address verbatim — the TCP data port carries the entire wire protocol (control, RDMA bootstrap, and inline-bulk codes for hosts without RDMA). There's no separate transport target; the client picks how to use it.

A complete reference copy ships at deploy/examples/client-config.yaml.

Field-by-field

Transport selection

memkv-client speaks two data transports, RDMA and TCP, and both are fully supported. RDMA delivers the lowest latency where an RDMA NIC and a clean fabric are available. TCP is a first-class high-throughput path, not a degraded shim, for the many environments where RDMA isn't viable: routed or multi-hop networks, cloud, mixed NICs. transport controls which one the client uses:

• auto (default): try RDMA on the configured rdma_devices first, fall back to TCP if no RDMA NIC is present at boot or any in-flight RDMA op fails. The fallback latches once — subsequent requests go straight to TCP. macOS / hosts without an HCA land in TCP from startup. The downgrade is logged at WARN.
• rdma: strict. The client errors at startup if rdma_devices is empty or the build is non-Linux (macOS / arm64 dev box). In-flight RDMA failures propagate instead of silently switching transports — you get a clear error pointing at the misconfiguration. Use this in production where RDMA must be the actual data path.
• rdma-rc: strict RDMA like rdma, but forces the RC data path even on DC-capable mlx5 hardware. RC is already the only path on non-mlx5 NICs (e.g. Broadcom bnxt_re), so this value is only needed to exercise the RC path on a fabric where DC would otherwise win, or as an ops escape hatch if DC misbehaves. Batches ride the RC control channel and the server serializes the wire stage per connection.
• tcp: skip RDMA entirely; no QP setup. The configured server address is the TCP target directly. To sustain throughput on fast links the client opens num_connections sockets per server and pipelines batched reads and writes across them, so TCP is a production-viable transport — not only for co-located dev hosts (e.g. Mac mini) but for any deployment where RDMA can't reach end to end.

The server's TCP listener carries the RDMA bootstrap (Connect) and, for clients whose RDMA data path is not active, the inline-bulk codes (TcpPut / TcpGet / TcpDelete + Exists). Clients on the RDMA path move every other control message onto RC SEND/RECV after bootstrap. Transport choice is the client's decision.

Port convention. The server binds the data port as TCP; clients use the configured host:port directly. The admin endpoint auto-binds at port + 1 with the same host IP — no separate knob.

Authentication

Every wire message between client and server is signed with HMAC-SHA256 over (header || ts_ns) and verified on receipt. The shared key is the auth_key field above (or the MEMKV_AUTH_KEY env override). Drift window is ±60 s — keep clocks in sync with chrony / ntpd. There is no unauthenticated mode; the client refuses to start without a key, and the server refuses to start without one either (loopback-only deployments are an exception: both sides derive a deterministic dev key from $HOME).

Generate a key with openssl rand -hex 32. The same value goes on both sides — server (network.auth_key in /etc/memkv/config.yaml or MEMKV_AUTH_KEY) and every client (yaml auth_key: here, or MEMKV_AUTH_KEY env). Keys never appear in logs.

License

memkv-client verifies a license at startup. The lookup chain is the yaml license: field → MEMKV_LICENSE → AISTOR_LICENSE → MINIO_LICENSE → ./minio.license → $HOME/.<binary_name>/minio.license.

Accepted plans: Free, Enterprise, EnterpriseLite, EnterprisePlus.

Getting a license. All MemKV licenses — Free and paid — are issued by MinIO SUBNET. Visit min.io/pricing to request a Free-tier license for evaluation or non-production use, or to start an Enterprise / EnterpriseLite / EnterprisePlus conversation with the MinIO team. The license file you receive is a JWT; point license: (or MEMKV_LICENSE) at either the inline token or the file path.

Environment variables

A flat index of every MEMKV_* variable. Client resolution is defaults → MEMKV_CONFIG yaml → env vars (env wins). The server reads /etc/memkv/config.yaml plus the two overrides below.

Server

AISTOR_LICENSE is consulted next, then MINIO_LICENSE for compatibility with the broader MinIO stack — neither is specific to MemKV.

LD_PRELOAD shim

Every client variable on this page applies to the LD_PRELOAD shim as well: those in the Client configuration table above, together with the client-scoped variables under Advanced tuning and Observability below. One variable applies only to the shim:

memkv admin

Advanced tuning

These are read once on first use and cached for the lifetime of the process — changing them at runtime has no effect.

Observability

Two client-side env vars enable optional telemetry exporters. They are also documented in Monitoring.

Debug