Troubleshooting Runbook

Start here during an incident. This page lists the first commands to run, what to collect before you open a support case, and the page to read for each symptom.

First commands

Run these commands first. They show whether the cluster is healthy, which components are degraded, and which errors clients receive.

Command What it shows
mc admin info ALIAS Which nodes and drives are online or offline, the erasure set layout, capacity, and the version on each node
mc admin trace --errors --verbose ALIAS The failing requests as the server sees them, including the S3 error code
Healthcheck probe Whether each node reports itself live and ready, independent of the load balancer

Use the S3 error code from the trace, not only the HTTP status that the client reports. For example, a 403 can be AccessDenied, SignatureDoesNotMatch, or RequestTimeTooSkewed, and each has a different cause.

Before you open a support case

Collect the following so that support can start work right away:

  1. A diagnostic report. Run mc support diagnostics ALIAS. If the machine that runs the command cannot reach SUBNET, add --airgap and attach the file it writes with mc support upload.
  2. When it started and what changed. The start time in UTC, and any upgrade, restart, configuration change, or traffic change around that time.
  3. The scope. One bucket or all buckets, one client or all clients, one node or the whole deployment.
  4. The exact error. Copy it from mc admin trace or the server log.

Support may later ask for the data and metadata of one object. See mc support inspect.

By symptom

Symptom Where to go
Drive failed, quorum lost Recover after a drive failure
Node down or unreachable Recover after a node failure
Site lost, failover to DR Recover after a site failure
Healing slow or not progressing Healing
TLS handshake or certificate errors TLS troubleshooting
Certificate expiring Certificate rotation
Bucket replication lagging or failing Bucket replication
Site replication out of sync Site replication
Batch replication job stuck Batch replication
Throughput lower than expected Network performance testing, performance tuning
RDMA fabric problems Troubleshoot RDMA
Slow or failing requests, API errors Debugging and troubleshooting
Drive usage higher than the buckets account for System path growth
Capacity running out Scaling
No alerts or metrics for these conditions Monitoring, alerts