CLI Reference

The memkv binary contains every operator command: drive setup, the server, the bundled documentation, the admin client, offline drive maintenance, benchmarks, and network checks.

memkv [GLOBAL OPTIONS] <COMMAND> [ARGS]

Commands:
  setup    Initialize drives and generate config (outputs YAML to stdout)
  start    Start the server
  doc      Serve the bundled MemKV documentation over HTTP
  admin    Query the admin HTTP endpoint of one or more running MemKV servers
  drive    Offline drive maintenance (server stopped); see the `drive` section
  bench    Benchmark the data path of a running server over the network
  net      Check the RDMA fabric: negotiated parameters, connectivity, PFC counters

Global

memkv <command> --help prints the help for one command.

setup

Formats the listed NVMe drives for MemKV and prints a working config.yaml to stdout. Logs go to stderr, so you can redirect stdout straight to the configuration file.

sudo memkv setup \
  --drives /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 \
  --rdma mlx5_0 \
  | sudo tee /etc/memkv/config.yaml

Behavior

• Logs each drive's model, optimal I/O size, and max_sectors_kb to stderr before formatting.
• Sets memory.max_size to 50% of available system memory, with a minimum of 1 GB.
• Without --force, stops at the first drive that already holds MemKV data and names it.
• Always writes storage.mode: direct. For the file or memory modes, edit the YAML by hand. See Configuration.

Exit codes

start

Runs the MemKV server. At startup it checks the license, starts RDMA, recovers stored blocks, and opens the data and admin listeners.

sudo memkv start --config /etc/memkv/config.yaml

Behavior

• Refuses to start on Linux kernels older than 6.8, because the io_uring and RDMA features MemKV uses are unstable on older kernels.
• Checks that the memory pool fits in free hugepages or in 80% of available system memory. If it does not, it prints a fix such as sysctl -w vm.nr_hugepages=….
• Starts RDMA on the device in rdma.device. If the device is missing or RDMA setup fails, the server logs an error, sets memkv_rdma_active=0, and keeps running without RDMA. This is acceptable for development, never for production.
• Recovers stored blocks from the configured drives. If there are none, it logs this and continues.
• On SIGINT or SIGTERM, writes its storage metadata to disk and exits.

Exit codes

doc

Serves this documentation site from inside the binary, so you can read it without internet access.

memkv doc

The command listens on a random loopback port and logs the URL at startup (http://127.0.0.1:<port>/).

admin

Queries the admin HTTP endpoint of one or more running MemKV servers. All routes start with /v1/.

memkv admin [--servers <URLS>] <subcommand> [SUBCOMMAND ARGS]

Server resolution

memkv admin uses the first of these that is set:

1. --servers <URLS> (short -s): a comma-separated list. Spaces and empty entries are ignored.
2. The MEMKV_SERVERS environment variable, in the same format.
3. http://127.0.0.1:9901.

An entry without a scheme, such as host:port, uses http://. For servers with TLS (network.tls_cert and network.tls_key), use https:// URLs.

export MEMKV_SERVERS="coe02:9901,coe04:9901"
memkv admin status

Subcommands

admin status flags

admin drives failed flags

admin drives reset flags

Admin HTTP API

memkv admin uses the HTTP endpoints that every server serves at 0.0.0.0:9901 (the data port + 1). The endpoints use HTTPS when network.tls_cert and network.tls_key are set, and HTTP otherwise. Tools that do not have the memkv binary, such as Helm hooks or external monitors, can call the endpoints with curl.

curl http://localhost:9901/v1/drives/failed

[
  {
    "device_id": 2,
    "failure_count": 3,
    "first_failed_at": 1738800000,
    "last_failed_at": 1738810000,
    "blacklisted": false
  }
]

A reset discards the drive's contents and sets up its metadata again. The drive then returns to service. To also write zeros over the data area, add ?secure=true; this takes time in proportion to the drive's size.

curl -X POST http://localhost:9901/v1/drives/2/reset
curl -X POST 'http://localhost:9901/v1/drives/2/reset?secure=true'

Exit codes

After an error, the command still queries the remaining servers, reports every failure, and sets the exit code at the end.

drive

Offline drive maintenance. These commands open a drive directly, so stop the server first. To reset a drive while the server runs, use memkv admin drives reset <device_id>.

drive reset

memkv drive reset <PATH> [--secure]

Erases the drive's metadata (superblock, journal, B+ tree, and bitmap) and sets it up again, and discards the data area (BLKDISCARD). All data on the drive is lost. The command reads the drive's device_id and the number of drives from the drive itself, so you do not need to know them. If the main superblock is unreadable, it uses the backup copy.

sudo memkv drive reset /dev/nvme2n1
sudo memkv drive reset /dev/nvme2n1 --secure

drive reset does not clear a blacklist. See the callout under Admin HTTP API.

drive verify

Checks the integrity of one drive. It only reads, never writes.

memkv drive verify <PATH> [--quick] [--json]

It checks, in order:

• the superblock and its backup copy at the end of the drive;
• every B+ tree page against its stored checksum;
• the journal: if the drive was not shut down cleanly, the command replays it, as the server does, so a drive after a crash is not reported as corrupt;
• the allocation bitmap against the slots the index uses, reporting wasted slots and missing entries;
• unless --quick is set, the CRC32 of every stored block. This is the only check that finds corrupted data, because the server does not verify checksums on read by default.

sudo memkv drive verify /dev/nvme2n1
sudo memkv drive verify /dev/nvme2n1 --quick --json

The exit code is 0 when the drive is clean and 1 when errors remain. The message then points you to drive fsck --repair.

drive fsck

Runs the same checks as verify. Without --repair, it only reports what it would fix (like xfs_repair -n). With --repair, it fixes the metadata.

memkv drive fsck <PATH> [--repair] [--quick] [--json]

A repair rebuilds the B+ tree from the valid entries, updates the bitmap to match, resets the journal, and marks the drive as cleanly shut down.

sudo memkv drive fsck /dev/nvme2n1            # dry run — report only
sudo memkv drive fsck /dev/nvme2n1 --repair   # apply repairs

drive health

Shows a drive's model, capacity, fill level, whether it was shut down cleanly, and its recorded failures. It reads only metadata, not the data area.

memkv drive health <PATH> [--json]

sudo memkv drive health /dev/nvme2n1
sudo memkv drive health /dev/nvme2n1 --json

drive bench

Benchmarks the storage layer on its own, with no network and no running server. It can test raw NVMe drives or file-based storage.

memkv drive bench <SUITE> [flags]

To test raw NVMe drives, pass --devices /dev/nvme0n1,.... To test file-based storage, pass a scratch directory with --dir. Results print to stdout; with --output, they are also written as JSON.

sudo memkv drive bench throughput --devices /dev/nvme0n1 --data-sizes 4096,65536,1048576
memkv drive bench full --dir /tmp/bench --output ./benchmark-results

bench

Benchmarks the whole data path: it sends PUT and GET requests to a running MemKV server over the network, through client, RDMA or TCP, server, and storage. To test storage alone, use memkv drive bench. To check the fabric without load, use memkv net.

MEMKV_AUTH_KEY=<hex> memkv bench --servers <ip:port>[,...] [--devices mlx5_0,...] [flags]

Common flags (memkv bench --help lists all of them):

--auth-key (or MEMKV_AUTH_KEY) is required and must match the server's network.auth_key.

MEMKV_AUTH_KEY=$KEY memkv bench --servers 10.0.1.1:9900 --devices mlx5_0 --block-size 65536 --threads 8
MEMKV_AUTH_KEY=$KEY memkv bench --tcp --servers 10.0.1.1:9900 --write-only

net

Checks the RDMA fabric on RoCEv2 and InfiniBand without generating load. Use it to confirm a NIC or fabric works, before or without a running server.

net show

Prints the settings MemKV would use on a device: the port's link layer, path MTU, whether the DC transport is available, and the fabric settings. On Ethernet these are the detected RoCEv2 GID index and traffic class (DSCP). On InfiniBand they are the LID and partition index. It makes no connection.

It also prints the NIC's kernel driver and vendor, whether the host's libibverbs can register GPU memory as a dma-buf, which needs rdma-core 34 or later, and the kernel peer-memory clients registered with the RDMA core, such as amdkfd or nv_mem, which register GPU memory without one. Whether the NIC driver accepts a given registration is known only by trying one, which memkv bench --gpu-devices does.

memkv net show [--device mlx5_0] [--port 1] [--gid-index N] [--pkey-index N] [--traffic-class N] [--service-level N] [--mtu 4096]

On RoCEv2, 0 means detect automatically and any other value sets the setting:

• --gid-index is detected from the port's GID table (the routable RoCEv2 entry).
• --traffic-class and --service-level are detected together from the NIC's DCB settings. When --traffic-class is 0 and detection succeeds, it also sets the service level and ignores any --service-level you pass. Set both or neither.

On InfiniBand, --gid-index and --traffic-class are ignored, and --service-level 0 is not automatic detection: it selects virtual lane 0. --pkey-index selects the InfiniBand partition; 0 is the default partition. --mtu must be 256, 512, 1024, 2048, or 4096.

net verify

Connects to a running MemKV server over RDMA, reports the transport in use (DC or RC) and the fabric settings, then writes and reads back data to prove that it moves. Use it to check connections between NIC vendors, such as a Broadcom bnxt_re client and an NVIDIA mlx5 server, and to confirm an InfiniBand path end to end.

MEMKV_AUTH_KEY=<hex> memkv net verify --server <ip:port> [--device mlx5_0] [fabric flags]

It takes the same fabric flags as net show, plus --auth-key (or MEMKV_AUTH_KEY), which must match the server's key.

net probe

Takes a snapshot of a NIC's priority, pause, and PFC counters (from ethtool -S). Compare snapshots from before and after a load test to confirm that RoCE traffic used the lossless priority. The traffic class alone does not prove that the fabric kept the traffic lossless. RoCEv2 only: InfiniBand has its own flow control and no such counters.

memkv net probe [--device mlx5_0] [--netdev <iface>]

On a NIC with several ports, where the device-to-interface mapping is unclear, pass --netdev to name the interface.

net topo

Shows how far each RDMA NIC is from each GPU, and names the NIC each GPU should use for RDMA.

memkv net topo [--devices mlx5_0,mlx5_1] [--gpu-devices 0,1]

GPUDirect RDMA reaches line rate only when the NIC and the GPU are on the same PCIe switch, shown as PIX in the output (the same term nvidia-smi topo -m uses; rocm-smi --showtopo shows the same layout on AMD). Any other path goes through the CPU. The right memkv_scratch_medium setting depends on this result, and the vLLM plugin makes the same check at startup to choose each rank's NIC, so run this command to see what the plugin will choose. By default it reports every device under /sys/class/infiniband and every GPU visible to CUDA or HIP, and prints which runtime it found. The verdict for the whole host reads every NVIDIA and AMD GPU the drivers list, whatever the visibility mask hides.

Environment variables

Default ports

The data port is a TCP port. It carries control messages, RDMA setup, and TCP data. RDMA data does not use it: RoCEv2 uses UDP port 4791, and InfiniBand does not use IP ports. The admin endpoint is always at the data port + 1. On macOS, the server has no RDMA, but it listens on the same network.address, so clients use the same address as on Linux.