Monitoring

The MemKV server serves health endpoints and Prometheus metrics on its admin HTTP endpoint (default: 0.0.0.0:9901). The MemKV client has its own metrics exporters, which are off by default.

Client-side exporters

The MemKV client (the NIXL plugin and the LD_PRELOAD shim) reports request latency, transport state, and batch size. To turn the metrics on, set one of these environment variables. If neither is set, the client collects no metrics.

Each process starts one exporter. If MEMKV_OTEL_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT is set, the client uses OTLP and does not start the Prometheus endpoint. Because the published packages do not include OTLP export, use MEMKV_PROMETHEUS_BIND, and make sure neither OTLP variable is set in the client's environment.

Health endpoints

All admin endpoints are versioned under /v1/.

curl http://localhost:9901/v1/health
# {"status":"healthy","rdma_active":true,"drives_online":12,"drives_offline":0,"drives_total":12}

status is healthy, degraded, or unhealthy. The HTTP status code matches: 200 for healthy, 503 for unhealthy.

Status endpoint

curl http://localhost:9901/v1/status

{
  "memory": {
    "used_bytes": 1073741824,
    "total_bytes": 265289728000,
    "slabs_allocated": 512,
    "slabs_total": 126464
  },
  "storage": {
    "num_devices": 24,
    "block_size_bytes": 4194304,
    "total_blocks": 1000,
    "raw_bytes_total": 384000000000000,
    "usable_bytes_total": 383979687936000,
    "used_bytes_total": 176160768,
    "devices": [
      {
        "device_id": 0,
        "path": "/dev/nvme0n1",
        "total_slots": 3815461,
        "allocated_slots": 42,
        "raw_capacity_bytes": 16000000000000,
        "usable_bytes": 15999153635328,
        "metadata_bytes": 846364672,
        "used_bytes": 176160768
      }
    ]
  },
  "rdma": {
    "active_connections": 8
  },
  "blocks_total": 1000,
  "blocks_persisted": 950,
  "uptime_seconds": 3612
}

Prometheus metrics

Request operations

RDMA transport

TCP transport

The latency histograms do not include time spent waiting on the network throttle. They measure only the server's own work.

Network throttle

The server always exports these metrics. When network.throttle is off, they stay at zero.

Useful queries:

# Is the cap binding right now? Seconds of delay per second of wall time.
rate(memkv_throttle_delay_seconds_total[1m])

# Achieved bulk rate in bits/s, to compare against your configured cap.
rate(memkv_throttle_bytes_total[1m]) * 8

# Mean delay per throttled transfer.
rate(memkv_throttle_delay_seconds_total[5m])
  / rate(memkv_throttle_waits_total[5m])

If latency is high and the first query stays at zero, the throttle is not the cause. Check the drives.

Storage devices

Capacity reclaim

What to watch:

• Alert when memkv_storage_eviction_starved_passes_total rises. Storage is above its high watermark and nothing is left to reclaim, so writes will soon fail. Add capacity, or find what is holding the blocks.
• Fill ratio: memkv_storage_used_bytes / memkv_storage_usable_bytes. Normally it moves between the two watermarks. If it stays at the high watermark, reclaim cannot keep up with writes.
• memkv_storage_evictions_unindexed_total rising steadily: the server is reclaiming blocks oldest first, because they were stored before the last restart and have not been read since. This is expected for a while after a restart on large drives, and it slows as clients read their working set. If it does not slow, the server is reclaiming blocks without regard to which ones are in use.
• Passes per eviction: if memkv_storage_eviction_passes_total rises much faster than memkv_storage_evictions_total, reclaim passes run but find little. Widen the gap between the watermarks.

Memory and blocks

Admin API

Storage pipeline (hot-path tracing)

Detailed timings inside the write path. Use them to diagnose slow writes under sustained load.

Timings for each request stage, labeled op and step:

The client metric memkv_client_step_seconds uses the same op and step labels. To find where a slow put spends its time, compare memkv_server_step_seconds{op="commit",step="total"} (time on the server) with memkv_client_step_seconds{op="put",step="commit"} (time including the network).

Kubernetes probes

livenessProbe:
  httpGet:
    path: /v1/health
    port: 9901
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /v1/ready
    port: 9901
  initialDelaySeconds: 5
  periodSeconds: 5