Troubleshooting Runbook

Start here during an incident. This page lists the first commands to run, what to collect before you open a support case, and the page to read for each symptom.

First commands

Run these commands first. They show whether the cluster is healthy, which components are degraded, and which errors clients receive.

CommandWhat it shows
mc admin info ALIASWhich nodes and drives are online or offline, the erasure set layout, capacity, and the version on each node
mc admin trace --errors --verbose ALIASThe failing requests as the server sees them, including the S3 error code
Healthcheck probeWhether each node reports itself live and ready, independent of the load balancer

Use the S3 error code from the trace, not only the HTTP status that the client reports. For example, a 403 can be AccessDenied, SignatureDoesNotMatch, or RequestTimeTooSkewed, and each has a different cause.

Before you open a support case

Collect the following so that support can start work right away:

  1. A diagnostic report. Run mc support diagnostics ALIAS. If the machine that runs the command cannot reach SUBNET, add --airgap and attach the file it writes with mc support upload.
  2. When it started and what changed. The start time in UTC, and any upgrade, restart, configuration change, or traffic change around that time.
  3. The scope. One bucket or all buckets, one client or all clients, one node or the whole deployment.
  4. The exact error. Copy it from mc admin trace or the server log.

Support may later ask for the data and metadata of one object. See mc support inspect.

By symptom

SymptomWhere to go
Drive failed, quorum lostRecover after a drive failure
Node down or unreachableRecover after a node failure
Site lost, failover to DRRecover after a site failure
Healing slow or not progressingHealing
TLS handshake or certificate errorsTLS troubleshooting
Certificate expiringCertificate rotation
Bucket replication lagging or failingBucket replication
Site replication out of syncSite replication
Batch replication job stuckBatch replication
Throughput lower than expectedNetwork performance testing, performance tuning
RDMA fabric problemsTroubleshoot RDMA
Slow or failing requests, API errorsDebugging and troubleshooting
Drive usage higher than the buckets account forSystem path growth
Capacity running outScaling
No alerts or metrics for these conditionsMonitoring, alerts

On this page