Monitoring

MemKV exposes health endpoints and Prometheus metrics on the admin HTTP endpoint (default: 0.0.0.0:9901). The client library ships separate exporters that are off by default.

Client-side exporters

memkv-client (the NIXL plugin and LD_PRELOAD shim) emits its own metrics — request latencies, transport state, batch sizing. Pick a sink by setting one of these env vars; with neither set, observability is disabled and no work is done in the hot path.

If both are set the OTLP path wins. The exporter is installed once per process — subsequent calls are no-ops.

Health endpoints

All admin endpoints are versioned under /v1/.

curl http://localhost:9901/v1/health
# {"status":"healthy","rdma_active":true,"drives_online":12,"drives_offline":0,"drives_total":12}

status is one of healthy, degraded, or unhealthy. HTTP status code mirrors the body — 200 for healthy, 503 for unhealthy.

Status endpoint

curl http://localhost:9901/v1/status

{
  "memory": {
    "used_bytes": 1073741824,
    "total_bytes": 265289728000,
    "slabs_allocated": 512,
    "slabs_total": 126464
  },
  "storage": {
    "num_devices": 24,
    "block_size_bytes": 4194304,
    "total_blocks": 1000,
    "raw_bytes_total": 384000000000000,
    "usable_bytes_total": 383979687936000,
    "used_bytes_total": 176160768,
    "devices": [
      {
        "device_id": 0,
        "path": "/dev/nvme0n1",
        "total_slots": 3815461,
        "allocated_slots": 42,
        "raw_capacity_bytes": 16000000000000,
        "usable_bytes": 15999153635328,
        "metadata_bytes": 846364672,
        "used_bytes": 176160768
      }
    ]
  },
  "rdma": {
    "active_connections": 8
  },
  "blocks_total": 1000,
  "blocks_persisted": 950,
  "uptime_seconds": 3612
}

Prometheus metrics

Request operations

RDMA transport

TCP transport

Neither latency histogram includes time spent waiting on the network throttle — they measure storage work, so pacing is subtracted out.

Network throttle

These are always exported. When network.throttle is off they stay flat at zero.

Useful queries:

# Is the cap binding right now? Seconds of delay per second of wall time.
rate(memkv_throttle_delay_seconds_total[1m])

# Achieved bulk rate in bits/s, to compare against your configured cap.
rate(memkv_throttle_bytes_total[1m]) * 8

# Mean delay per throttled transfer.
rate(memkv_throttle_delay_seconds_total[5m])
  / rate(memkv_throttle_waits_total[5m])

If latency is up and the first query is flat at zero, the cap is not the cause — look at the drives.

Storage devices

Capacity reclaim

What to watch:

• memkv_storage_eviction_starved_passes_total rising is the one to alert on. Storage is above its watermark and has nothing left to reclaim, so writes are heading for failure. Add capacity or find what is holding blocks.
• Fill ratio — memkv_storage_used_bytes / memkv_storage_usable_bytes. Under normal operation it oscillates between the two watermarks. Sitting pinned at the high watermark means reclaim can't keep up with writes.
• memkv_storage_evictions_unindexed_total rising steadily means the node is reclaiming without recency, oldest-first, because those blocks predate the last restart and haven't been read since. Expected for a while after a restart on large drives; it should taper as the working set gets touched. If it doesn't, the cache is being reclaimed with no regard for what's hot.
• Passes per eviction — memkv_storage_eviction_passes_total climbing much faster than memkv_storage_evictions_total means passes are running and finding little. Widen the gap between the watermarks.

Memory and blocks

Admin API

Storage pipeline (hot-path tracing)

Fine-grained timings inside the write pipeline — useful for diagnosing sustained-write behavior under load.

Cut by request stage, labelled op and step:

These pair with the client's memkv_client_step_seconds, which uses the same op and step cut. Comparing memkv_server_step_seconds{op="commit",step="total"} against memkv_client_step_seconds{op="put",step="commit"} splits a slow put into time on the wire versus time under the server's persist path.

Kubernetes probes

livenessProbe:
  httpGet:
    path: /v1/health
    port: 9901
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /v1/ready
    port: 9901
  initialDelaySeconds: 5
  periodSeconds: 5