Debugging and Troubleshooting

Use these tools and techniques to debug MinIO AIStor deployments.

HTTP tracing

Use mc admin trace to capture real-time HTTP API calls to and from the MinIO AIStor server.

Default trace shows only API operation names and HTTP response status:

mc admin trace ALIAS

Trace entire HTTP requests including headers and body:

mc admin trace --verbose ALIAS

Trace all HTTP requests including internode communication:

mc admin trace --all --verbose ALIAS

Trace call types

The --call flag selects which category of operations to trace:

Call type	Alias	Description
`s3`		S3 API calls (default)
`internal`		Internal RPC and internode calls
`storage`		Storage I/O calls
`os`		Operating system calls
`scanner`		Background scanner operations
`healing`	`heal`	Healing and recovery operations
`ilm`		Lifecycle management operations
`bootstrap`		Server startup operations
`formatting`	`fmt`	Drive formatting operations
`ftp`		FTP protocol operations
`tables`		AIStor Tables (Iceberg) API calls
`decommission`	`decom`	Pool decommissioning
`rebalance`		Server pool rebalancing
`replication-resync`	`resync`	Replication resynchronization
`batch-replication`	`brep`	Batch replication operations
`batch-keyrotation`	`brot`	Batch key rotation operations
`batch-expiration`	`bexp`	Batch object expiration

Combine multiple call types with a comma:

mc admin trace --call s3,internal,storage ALIAS

Filter by API operation

Use --funcname to trace specific S3 API operations:

mc admin trace --call s3 --funcname s3.PutObject ALIAS

Combine multiple operations with a comma-separated list:

mc admin trace --call s3 --funcname s3.GetObject,s3.HeadObject ALIAS

Filter by path

Use --path to trace operations on a specific bucket or object:

mc admin trace --call s3 --path "mybucket/path/to/object" ALIAS

Filter by request or response size

Use --filter-request or --filter-response with --filter-size to trace only large operations:

mc admin trace --call s3 --filter-request --filter-size 100MB ALIAS

Filter by HTTP status code

Use --status-code to trace only errors:

mc admin trace --call s3 --status-code 403,500,503 ALIAS

Performance profiling

Use mc support profile to capture CPU, memory, and blocking profiles during a performance issue. Run the profile during the slow period for the most useful data.

mc support profile ALIAS --type cpu,mem,block

Additional profile types: mutex, trace, threads, goroutines, cpuio, runtime, metrics.

Reading profile output

Analyze the downloaded profile files with go tool pprof:

go tool pprof -top profile-NODE-cpu.pprof

Common bottleneck patterns:

Profile pattern	Cause	Action
High time in `syscall.openat` / `xlStorage.readMetadata`	Metadata I/O saturation from many objects	Enable drive cache (`MINIO_CACHE_ENABLE=on`), upgrade for metadata path optimizations
High time in `runtime.selectgo` / `sync.Mutex.Lock`	Lock contention under high concurrency	Upgrade for lock contention fixes, check internode latency
High `xlStorage.WalkDir`	Listing operations scanning large prefixes	Reduce prefix depth, check scanner delay setting
`runtime.newosproc` in crash dump	OS thread exhaustion	Increase `ulimit -u` or container pids limit

Drive cache metrics

When drive caching is enabled (MINIO_CACHE_ENABLE=on), monitor cache effectiveness with Prometheus:

Cache utilization:

sum(minio_system_drive_cache_used) / sum(minio_system_drive_cache_capacity)

Cache hit ratio:

sum(minio_system_drive_cache_hits) / (sum(minio_system_drive_cache_hits) + sum(minio_system_drive_cache_misses))

Multipart upload troubleshooting

Multipart uploads can stall or accumulate if clients disconnect before completing the upload.

Trace multipart operations

Trace the full lifecycle of multipart uploads in real time:

mc admin trace --call s3 --funcname s3.NewMultipartUpload,s3.PutObjectPart,s3.CompleteMultipartUpload ALIAS

Find large parts being uploaded:

mc admin trace --call s3 --funcname s3.PutObjectPart --filter-request --filter-size 100MB ALIAS

List incomplete multipart uploads

mc ls ALIAS/BUCKET --incomplete --recursive

Clean up stale multipart uploads

Remove all incomplete multipart uploads older than a specified duration:

mc rm ALIAS/BUCKET --incomplete --recursive --older-than 7d --force

Automate multipart cleanup with lifecycle rules

Set a lifecycle rule to automatically abort incomplete multipart uploads after a number of days:

mc ilm rule add ALIAS/BUCKET --expire-days 7

This prevents stale multipart uploads from accumulating and consuming disk space.

SUBNET health diagnostics collect system information to help ensure the underlying infrastructure is configured correctly. This test is resource-intensive and is recommended at initial deployment and when investigating failure scenarios.

mc support diagnostics ALIAS

The command collects:

Admin info
CPU information
Disk hardware details
OS information
Memory information
Process information
Configuration
Drive performance
Network performance

The output is saved as a compressed JSON file.

The diagnostics output file may contain sensitive information about your environment. Inspect the contents before sharing on any public forum.

Metadata inspection

MinIO AIStor stores object metadata in xl.meta files using a binary format. Use the xl-meta tool to decode these files.

Install xl-meta

Install the xl-meta tool using Go:

go install github.com/minio/minio/docs/debugging/xl-meta@latest

Use xl-meta

Run xl-meta in a directory containing xl.meta files:

xl-meta ./xl.meta

Decode multiple files recursively:

xl-meta ./**/xl.meta

View inline data stored in metadata:

xl-meta -data xl.meta

Export inline data to a file:

xl-meta --export xl.meta

Remote backend inspection

mc support inspect collects files from all backend drives based on a path pattern. Matching files are collected in a zip file with their respective host, drive, and path information.

Collect xl.meta from a specific object:

mc support inspect ALIAS/bucket/path/to/file.txt/xl.meta

Collect all xl.meta files recursively:

mc support inspect ALIAS/bucket/path/**/xl.meta

Collect part files for all versions of an object:

mc support inspect ALIAS/bucket/path/to/file.txt/*/part.*

The xl-meta tool accepts zip files as input and outputs all xl.meta files found within the archive:

xl-meta inspect.6f96b336.zip

Encrypted inspection

mc support inspect encrypts its output by default, producing an encrypted zip file and a one-time decryption key. No flag is required:

mc support inspect ALIAS/bucket/path/to/file.txt/xl.meta

The decryption key is only shown once and cannot be recovered. The encrypted file can be safely shared without the decryption key.

To produce unencrypted output using the older export method, pass --legacy, which does not encrypt data.

Decryption tool

Install the decryption tool:

go install github.com/minio/minio/docs/debugging/inspect@latest

Decrypt an inspection file:

inspect -key=<decryption-key> inspect.ad2b43d8.enc

If --key is not specified, an interactive prompt asks for it. The file name contains the beginning of the key to verify the correct key is used.

API version startup errors

A node can fail to start with a fatal API version error after an upgrade or downgrade. These failures are triggered by the API version the cluster runs.

cluster API version could not be determined

cluster API version could not be determined: .minio.sys/api-version is missing or unreadable on all local drives and no node quorum could supply a valid version

The node cannot read a valid API version from its drives and cannot obtain one from a quorum of peers.

To recover:

Confirm that all nodes run the upgraded binary.
Remove the .minio.sys/api-version file on all drives.
Restart the cluster simultaneously.

server is older than the data on its drives

this AIStor server is older than the data on its drives. The cluster was upgraded to a newer AIStor release and downgrading to an older release is not supported. Restart the server using the newer AIStor release that last ran this cluster (on-disk cluster API version <N>, this binary supports up to <M>)

The drives hold a cluster API version newer than this binary can serve, which happens when you attempt to downgrade to an older release after the cluster API version has already bumped.

To recover, restart the server using the newer AIStor release that last ran the cluster. Downgrade is no longer supported once the API version has been bumped. To retain a rollback window on future upgrades, see Retain a rollback window with a bake time and Downgrading MinIO AIStor versions.

Troubleshooting runbook for where to start during an incident and what to collect before opening a support case

Debugging and Troubleshooting

On this page