Debugging and Troubleshooting

This guide covers tools and techniques for debugging MinIO AIStor deployments.

HTTP tracing

Use mc admin trace to capture real-time HTTP API calls to and from the MinIO AIStor server.

Default trace shows only API operation names and HTTP response status:

mc admin trace ALIAS

Trace entire HTTP requests including headers and body:

mc admin trace --verbose ALIAS

Trace all HTTP requests including internode communication:

mc admin trace --all --verbose ALIAS

SUBNET health diagnostics

SUBNET health diagnostics collect system information to help ensure the underlying infrastructure is configured correctly. This test is resource-intensive and is recommended at initial deployment and when investigating failure scenarios.

mc support diagnostics ALIAS

The command collects:

  • Admin info
  • CPU information
  • Disk hardware details
  • OS information
  • Memory information
  • Process information
  • Configuration
  • Drive performance
  • Network performance

The output is saved as a compressed JSON file.

The diagnostics output file may contain sensitive information about your environment. Inspect the contents before sharing on any public forum.

Metadata inspection

MinIO AIStor stores object metadata in xl.meta files using a binary format. Use the xl-meta tool to decode these files.

Install xl-meta

Install the xl-meta tool using Go:

go install github.com/minio/minio/docs/debugging/xl-meta@latest

Use xl-meta

Run xl-meta in a directory containing xl.meta files:

xl-meta ./xl.meta

Decode multiple files recursively:

xl-meta ./**/xl.meta

View inline data stored in metadata:

xl-meta -data xl.meta

Export inline data to a file:

xl-meta --export xl.meta

Remote backend inspection

mc support inspect collects files from all backend drives based on a path pattern. Matching files are collected in a zip file with their respective host, drive, and path information.

Collect xl.meta from a specific object:

mc support inspect ALIAS/bucket/path/to/file.txt/xl.meta

Collect all xl.meta files recursively:

mc support inspect ALIAS/bucket/path/**/xl.meta

Collect part files for all versions of an object:

mc support inspect ALIAS/bucket/path/to/file.txt/*/part.*

The xl-meta tool accepts zip files as input and outputs all xl.meta files found within the archive:

xl-meta inspect.6f96b336.zip

Encrypted inspection

Use --encrypt to produce an encrypted output file:

mc support inspect --encrypt ALIAS/bucket/path/to/file.txt/xl.meta

This outputs an encrypted file and a one-time decryption key.

The decryption key is only shown once and cannot be recovered. The encrypted file can be safely shared without the decryption key.

Decryption tool

Install the decryption tool:

go install github.com/minio/minio/docs/debugging/inspect@latest

Decrypt an inspection file:

inspect -key=<decryption-key> inspect.ad2b43d8.enc

If --key is not specified, an interactive prompt asks for it. The file name contains the beginning of the key to verify the correct key is used.