Configuration

MemKV has two configuration files:

• The server configuration, /etc/memkv/config.yaml. You rarely write it by hand: memkv setup detects the drives and NICs and prints a working config.yaml, which you then adjust. See Quick Start.
• The client configuration, a YAML file that the NIXL plugin and the LD_PRELOAD shim read from the path in MEMKV_CONFIG. See Client configuration.

Minimal config

Most deployments set only these fields. Every other field has a default that suits most deployments.

network:
  address: 0.0.0.0:9900 # data plane; admin auto-binds at port + 1

memory:
  block_size: 2 MiB
  max_size: 247 GiB # RDMA-pinned RAM (size to your box)

rdma:
  device: mlx5_0

storage:
  mode: direct # direct | file | memory
  block_size: 4 MiB
  drives:
    - { media: /dev/nvme0n1 }
    - { media: /dev/nvme1n1 }
    - { media: /dev/nvme2n1 }

This is a complete configuration for a typical bare-metal node. The sections below describe each field and when to change it.

Network

network:
  address: 0.0.0.0:9900
  # tls_cert: /etc/memkv/tls/fullchain.pem
  # tls_key:  /etc/memkv/tls/privkey.pem
  # auth_key: <64 hex chars, 32 bytes>
  throttle:
    enabled: false
    bits_per_sec: 200 Gbps
    burst_window_ms: 100

address: The TCP address the server listens on. Clients connect here for control messages, to set up RDMA, and for TCP data. RDMA data does not use this port. The admin HTTP(S) endpoint listens on the same IP at port + 1, so 0.0.0.0:9901 when address is 0.0.0.0:9900.

• On a host with several networks, set the IP to limit MemKV to one NIC.
• Change the port if 9900 or 9901 is already in use.
• When rdma.devices lists more than one NIC, keep the wildcard address 0.0.0.0. A specific IP makes every other NIC unreachable.

node_id: A stable name for this node. Clients use it to decide which node holds a key. A node that has several addresses, one per NIC, is still one node. The default is the hostname.

If you change node_id, clients no longer look for this node's keys under the old name. The keys stay on disk, but clients read them as misses and compute them again. Set node_id explicitly if your hostnames can change, and then do not change it.

network:
  address: 0.0.0.0:9900
  node_id: 10.0.0.1:9900 # the address clients configure for this node

tls_cert / tls_key: Set both to serve the admin endpoint over HTTPS. Use HTTPS when the admin port is reachable from outside a trusted network, for example by scrapes from another zone or through an ingress controller. The server refuses to start if only one is set.

auth_key: The shared key that signs every message between client and server with HMAC-SHA256. It is 64 hex characters (32 bytes). Required: the server refuses to start without one. The one exception is a server that listens only on loopback: it derives a development key from $HOME and logs a warning.

• Generate a key with openssl rand -hex 32.
• MEMKV_AUTH_KEY overrides this field. The Helm chart always reads the key from a Kubernetes Secret.
• Clients must use the same key.
• Clocks on clients and servers must agree within 60 seconds. Use chrony or ntpd.

Throttle

The throttle limits the rate of bulk data transfers on this server. It is off by default, and MemKV then uses the full rate of the network. Turn it on when MemKV shares a NIC with training or storage traffic that needs reserved bandwidth.

enabled: Turns the throttle on or off. When false, MemKV ignores the other fields. The server refuses to start if enabled is true and bits_per_sec is not set.

bits_per_sec: The rate limit. Accepts bps, Kbps, Mbps, Gbps, Tbps, or a number of bits per second without a unit.

• Prefixes are decimal, as on NIC and switch counters: 200 Gbps is 200,000,000,000 bits/s, about half of one 400GbE NIC.
• Units are not case-sensitive, and Gb, Gbit, and Gbps all mean gigabits per second. So 200 GB means 200 gigabits per second, not gigabytes.
• Byte units such as GiB and MiB are rejected at startup. To convert from bytes, multiply by 8: 25 GB/s of payload is 200 Gbps.
• Rates above 1000 Tbps are rejected, with or without a unit. A value that large is a units mistake.

One limit covers both RDMA and TCP, so a client that uses both cannot get twice its share. The limit covers the batch operations that the NIXL, LMCache, and SGLang integrations use.

The throttle does not limit control messages (Connect, Lookup, Exists, Commit, Delete) or the admin endpoint. Health, status, and metrics stay available when the data path is at its limit.

On clients that use RDMA, control messages and data transfers share the RDMA worker threads (rdma.num_worker_threads, default 64). With a low limit, every worker can be waiting, and control messages are delayed behind them. If control latency rises under a strict throttle, increase rdma.num_worker_threads. TCP clients are not affected.

burst_window_ms: How far ahead of the limit the server may send, in milliseconds. Default 100. The burst is available from the first transfer, so bursty inference traffic is not slowed into a sawtooth.

So the limit is a long-run average, not a hard limit for every interval. Any interval can exceed bits_per_sec by up to the burst window. At the default, one second can carry about 110% of the limit. At burst_window_ms: 2000, one second can carry three times the limit. Over longer periods, the rate settles at bits_per_sec. Choose the window from the shortest interval over which the limit must hold, or set 0 for a strict limit.

The burst window is the whole allowance: the server never gets further ahead than this, however long it has been idle. Values above 10,000 are rejected, because they are most likely seconds typed as milliseconds.

With a low limit, consider storage.block_size. At 1 Mbps, the default 100 ms window covers about 12 KB, so one 4 MiB block waits about 33 seconds. This is the throttle working as configured.

For RDMA transfers and TCP reads, the throttle waits before any data moves. For TCP writes, the data has already arrived when the server counts it, so the server delays its response instead. The average still matches the limit, but incoming traffic can briefly exceed the burst window, depending on how many requests the clients keep in flight.

The limit applies per server. Ten servers at 200 Gbps allow 2 Tbps in total, so divide your network budget by the number of servers.

To check whether the throttle is slowing transfers, query the rate of memkv_throttle_delay_seconds_total, not its value. The counter only increases, so its value stays above zero after the first delay.

rate(memkv_throttle_delay_seconds_total[1m])

A rate above zero while data is moving means transfers are waiting on the throttle. A rate of zero means the throttle is not the cause; check the drives. See Monitoring for all throttle metrics.

Memory

memory:
  block_size: 2 MiB
  max_size: 247 GiB

Sizes accept the units B, KiB, MB, MiB, GB, GiB, TB, and TiB, with or without a space.

max_size: The most important setting. This is the host memory that MemKV pins for RDMA and uses to stage all client I/O. Set it to about 75–85% of the free RAM on the host, after you subtract the OS, page cache, and other services. The default, 1 GiB, is for development only. Production deployments need hundreds of gigabytes.

block_size: The size of one block in this memory. The default, 2 MiB, suits typical RDMA transfers and rarely needs to change. The maximum number of blocks is max_size / block_size.

RDMA

MemKV runs on two RDMA fabrics: RoCEv2 on Ethernet and native InfiniBand. At startup, MemKV reads the port's link layer and uses the right addressing for that fabric. Some settings below apply to one fabric only; each says which. For fabric setup, see the RoCEv2 and InfiniBand runbooks.

rdma:
  device: mlx5_0
  port: 1
  gid_index: 0 # 0 = auto-detect the routable RoCEv2 GID
  pkey_index: 0 # InfiniBand partition; 0 = the default partition
  traffic_class: 0
  service_level: 0
  mtu: 4096
  cq_depth: 4096
  timeout: 14
  retry_count: 7
  num_dcis: 256
  dc_key: 0
  num_worker_threads: 64
  num_responder_threads: 16

The defaults suit a typical RoCE or InfiniBand cluster. Most deployments set only device or devices.

device: The RDMA NIC to use. mlx5_0 is the first NVIDIA (Mellanox) NIC. Set it on a host with several NICs to choose one. memkv setup detects it. When devices is set, MemKV ignores device.

devices: A list of RDMA NICs to serve on. On a GPU host where each GPU has its own NIC on its own subnet, a server on one NIC is reachable directly from only one GPU. The other GPUs must cross the CPU to reach it, which costs most of the bandwidth. List every NIC, and the server answers each client on the NIC its request arrived on.

rdma:
  devices: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]

All the NICs share one memory pool, so adding NICs does not use more memory. For clients to reach every NIC:

• Set network.address to the wildcard address (0.0.0.0:9900). Otherwise only the NIC that owns the address is reachable.
• Give each NIC an address on its own subnet.

The startup log lists each NIC with its addresses and warns about both mistakes. /v1/status reports the connections on each NIC.

port: The port on the NIC. Keep 1 unless you have a dual-port card, such as a ConnectX-7, and want MemKV on the second port. Port numbers count from 1 on each NIC.

gid_index: RoCEv2 only. The entry in the NIC's GID table that MemKV uses. 0 selects the routable RoCEv2 entry automatically. On mlx5, index 0 is a RoCEv1 entry, which has no IP header and so no DSCP. On a fabric that classifies traffic by DSCP, RoCEv1 traffic always uses the best-effort queue. Set a non-zero value to choose an entry yourself, for example with mixed RoCEv1 and RoCEv2 entries or VLAN subnets. If MemKV finds no RoCEv2 entry, it uses index 0 and logs a warning. InfiniBand ignores this setting.

pkey_index: InfiniBand only. The index in the port's P_Key table of the partition that MemKV joins. 0 is the default partition, which is correct on RoCEv2 and on an InfiniBand fabric without partitions. Set it when the node belongs to another partition: ports in different partitions cannot exchange data. If the entry at this index is empty, MemKV does not start RDMA and logs the index. The server keeps running without RDMA, and clients stop using that NIC, so check the startup log. MEMKV_RDMA_PKEY_INDEX overrides this field. See the InfiniBand runbook.

traffic_class: RoCEv2 only. The IP DSCP/ToS byte on outgoing traffic. 0 detects it from the NIC's DCB settings: MemKV finds the lossless PFC priority and the DSCP mapped to it, so RDMA uses the lossless queue your fabric defines. Set a non-zero value to choose it yourself: the ToS byte is DSCP << 2, for example 104 for DSCP 26. If detection finds nothing, the value stays 0 (best effort) and MemKV logs a warning. On InfiniBand, MemKV ignores this setting and logs that it is ignored. MEMKV_RDMA_TRAFFIC_CLASS overrides this field.

service_level: Applies to both fabrics, with different meanings.

• On RoCEv2, it is the service level (0–7) on outgoing traffic. On mlx5, the NIC queues traffic by service level, not by DSCP. So the service level must equal the lossless PFC priority, or RDMA uses the best-effort queue even with the right DSCP. 0 uses the priority detected with traffic_class.
• On InfiniBand, it selects a virtual lane through the subnet manager's SL-to-VL map, which is unrelated to Ethernet priorities. Do not copy a RoCEv2 value to an InfiniBand cluster.

MEMKV_RDMA_SERVICE_LEVEL overrides this field.

mtu: The RDMA path MTU. 4096 needs jumbo frames on every hop: NIC, switch, and peer. Performance increases with MTU, so use 1024 or 2048 only if your network cannot carry 4096. The value must not be larger than the port's active MTU; MemKV refuses to start if it is. Set the same value on the server and on clients (MEMKV_RDMA_MTU), and configure the network to match. See the RoCEv2 setup runbook.

cq_depth: The completion queue size. Increase it if the log shows "CQ overflow" under bursts of load.

timeout: The ACK timeout for each queue pair, as 4.096µs × 2^timeout. 14 is about 67 ms, which suits a LAN where packet loss is rare. Lower it (for example, 12 is about 16 ms) on very low-latency networks to detect failures sooner. Raise it on lossy or oversubscribed networks.

retry_count: The number of retransmissions before MemKV closes a connection. 7 is the maximum that RDMA allows. Use 3–4 for faster client failover.

num_dcis: The number of DC initiators (mlx5 only), which move data to and from clients. 256 is enough for most NICs. Increase it under very high concurrency.

dc_key: A 24-bit key that the server and clients share on the DC transport. 0 means no key. Set a value to keep MemKV clusters that share one RoCE network apart.

num_worker_threads: The number of threads that process client reads and writes. Increase it if these threads are busy all the time. If they are idle while requests wait, the bottleneck is elsewhere.

num_responder_threads: The number of threads that send responses to clients. Increase it only if sending responses becomes a bottleneck.

Storage

storage:
  mode: direct # direct | file | memory
  block_size: 4 MiB
  flap_threshold: 5
  drives:
    - media: /dev/nvme0n1
      max_size: 2 TiB # cap usage to a subset of the physical drive
    - media: /dev/nvme1n1 # max_size omitted → use the full device
  # max_size: 16 GiB           # only used when mode: memory (no drives)
  eviction:
    enabled: true
    high_watermark: 70
    low_watermark: 60

mode:

• direct (default): MemKV writes to raw NVMe drives, bypassing the file system. This is the fastest mode. Each block is on one drive. Linux only.
• memory: MemKV keeps data in RAM only, with no persistence. Use it for development, benchmarks, and short-lived caches. storage.max_size sets the capacity.
• file: MemKV stores data in regular files on any file system that supports mmap. Use it for single-node development, CI, or a GPU host with no spare NVMe drive. The file system limits the bandwidth, so use direct in production. Each drives entry is one file, created at its max_size (default 4 GiB).

block_size: The size of one slot on the drive. Each value gets its own slots, so a value much smaller than a slot wastes the rest of it. The default, 4 MiB, suits the multi-megabyte KV values of large models. If your values are much smaller, a smaller size holds more KV on the same drive. See Choosing storage.block_size to measure your value size, choose a slot size, and change it on an existing drive.

flap_threshold: The number of consecutive I/O failures after which MemKV takes a drive offline. 5 suits most deployments. Use 1–2 to remove a failing drive sooner. Raise it to keep an unreliable drive in service until you replace it. Ignored in file and memory modes.

drives[].media: In direct mode, the path of a raw NVMe device such as /dev/nvmeXn1. MemKV formats the drive on first use, so do not list OS drives. In file mode, the path of a regular file, which MemKV creates at max_size when it first opens it.

drives[].max_size: The capacity MemKV uses on this drive. Omit it to use the whole drive in direct mode, or 4 GiB in file mode. Use it to give MemKV only part of a large drive.

storage.max_size: The capacity for mode: memory. Ignored in direct and file modes.

eviction: MemKV is a cache, so it frees space on its own instead of filling up and refusing writes. Keep eviction on unless you have a reason to turn it off.

When storage reaches high_watermark percent full, MemKV removes the least recently used blocks until storage is below low_watermark. It TRIMs the freed space before reusing it.

• enabled: true by default. When it is off, the drives fill and then reject new keys until a client deletes some. Turn it off only when the client manages retention and you prefer failed writes to lost blocks.
• high_watermark: 70. Eviction starts here. The free space lets TRIM keep ahead of new writes. Must be below 95, where MemKV stops accepting new keys.
• low_watermark: 60. Eviction stops here. Must be below high_watermark, or the server refuses to start. The gap between the two sets how much each pass removes: a narrow gap makes eviction run almost all the time; a wide gap removes more of the cache at once.

MemKV evicts blocks it has read or written since startup in least recently used order. After a restart on large drives, blocks that no client has read yet have no usage history, so MemKV evicts them oldest first.

Logging

logging:
  level: info # trace | debug | info | warn | error
  format: json # json | text

level: Use info in production. Use warn or error to reduce log volume. Use debug or trace only to debug; trace produces a lot of output.

format: Use json for log collectors such as Loki, ELK, or Datadog. Use text to read logs directly.

Client configuration (MEMKV_CONFIG)

The NIXL plugin (libplugin_MEMKV.so) and the LD_PRELOAD shim (libmemkv_preload.so) read their settings from a YAML file. Set MEMKV_CONFIG to its path. MEMKV_* environment variables override fields in the file.

The client resolves each setting in this order, lowest priority first:

1. Built-in defaults (one local server, one rail, 256 MB cache, 8 connections)
2. MEMKV_CONFIG yaml file, if the env var is set
3. MEMKV_* env vars

# /etc/memkv/client.yaml — pointed at by `export MEMKV_CONFIG=...`
servers:
  - 10.0.0.1:9900
  - 10.0.0.2:9900
rdma_devices:
  - mlx5_0
  - mlx5_1
bind_addresses: # optional, one per server slot; ~ = the matched NIC's address
  - 192.168.1.10
  - 192.168.1.11
cache_size_mb: 256
connect_timeout_ms: 5000
gid_index: 0
pkey_index: 0 # InfiniBand partition; 0 = the default partition
num_connections: 8
license: /etc/memkv/minio.license
auth_key: <64 hex chars, 32 bytes>
# transport: auto       # rdma | rdma-rc | tcp; default auto

Each entry in servers is the server's network.address. The client connects to that address and chooses the transport itself.

The YAML file also accepts max_batch_size, staging_size_mb, and staging_slot_mb. See Advanced tuning.

Transport selection

The client supports two transports, RDMA and TCP, and both are fully supported. RDMA gives the lowest latency when the host has an RDMA NIC and a well-configured fabric. TCP is a high-throughput transport for networks where RDMA does not work end to end: routed or multi-hop networks, cloud, and mixed NICs. transport selects the transport:

• auto (default): The client tries RDMA on the rdma_devices first. It switches to TCP if no RDMA NIC is present at startup or if an RDMA operation fails, and then stays on TCP. Hosts without an RDMA NIC, including macOS, use TCP from the start. The client logs the switch as a warning.
• rdma: RDMA only. The client fails to start if rdma_devices is empty or the host is not Linux. RDMA errors are returned to the caller instead of switching to TCP, so a misconfiguration produces a clear error. Use this in production when RDMA must carry the data.
• rdma-rc: RDMA only, like rdma, but always uses the RC transport, even on mlx5 NICs that support DC. NICs other than mlx5, such as Broadcom bnxt_re, always use RC, so you do not need this setting for them. Use it if DC causes problems on mlx5.
• tcp: TCP only; the client does not set up RDMA. It opens num_connections sockets to each server and sends batched reads and writes across them in parallel. TCP is suitable for production in any deployment where RDMA cannot reach end to end, not only for development hosts such as a Mac mini.

Ports. Clients connect to the host:port in servers. The server's admin endpoint is always at port + 1 on the same IP.

Authentication

The client and server sign every message with HMAC-SHA256, using the shared auth_key, and check the signature on receipt. Their clocks must agree within 60 seconds, so keep them in sync with chrony or ntpd.

Authentication cannot be turned off. The client and the server each refuse to start without a key. The one exception is a deployment that uses only loopback addresses: there, each side derives a development key from $HOME. The keys match only when the client and server run with the same $HOME.

Generate a key with openssl rand -hex 32, and use the same value on both sides:

• Server: network.auth_key in /etc/memkv/config.yaml, or MEMKV_AUTH_KEY.
• Every client: auth_key in the client YAML file, or MEMKV_AUTH_KEY.

MemKV never writes keys to its logs.

License

The client checks for a license at startup. It reads every one of these sources, verifies each license it finds, and uses the newest valid one: the YAML license: field, MEMKV_LICENSE, AISTOR_LICENSE, MINIO_LICENSE, ./minio.license, and $HOME/.<binary_name>/minio.license. Each value can be the license itself or the path to a license file.

MemKV accepts Enterprise, EnterpriseLite, EnterprisePlus, and OEM (ENTERPRISE-OEM) licenses, all with the same binary. It has no free tier and rejects a FREE license.

Product. A license can name the product it was issued for. MemKV accepts a license issued for MemKV, or one that names no product. It rejects a license issued for another product, such as AIStor, on both the server and the client. You cannot share one license file across products; request a MemKV license from SUBNET.

Getting a license. MinIO SUBNET issues MemKV licenses. To buy Enterprise, EnterpriseLite, or EnterprisePlus, visit min.io/pricing. The license is a JWT. Set license: (or MEMKV_LICENSE) to the token or to the path of the file.

Environment variables

This section lists every MEMKV_* variable. For the client, environment variables override the MEMKV_CONFIG file, which overrides the defaults. The server reads /etc/memkv/config.yaml and the variables below.

Server

If MEMKV_LICENSE is not set, MemKV also checks AISTOR_LICENSE and then MINIO_LICENSE, which other MinIO products use. A license found there must still be valid for MemKV: a license issued for another product is rejected. See License.

LD_PRELOAD shim

The LD_PRELOAD shim reads every client variable on this page: those in the Client configuration table, and those under Advanced tuning and Observability. One variable applies only to the shim:

memkv admin

Advanced tuning

The client reads these variables once, when it first needs them. Changing them while the process runs has no effect.

Observability

Two client variables turn on metrics exporters. Monitoring also describes them.

Debug