Troubleshoot System Path Growth

MinIO AIStor keeps its own state under the reserved .minio.sys path on every drive. Most of that state is small and fixed, but a dozen or so paths hold transient content: upload staging, deferred deletes, cached listings, job reports, and credential records. Each of those paths has two sides: a writer that adds entries, and a background sweeper that removes them again. Growth means the two have fallen out of balance. Either the sweeper stalled, or the workload adds entries faster than the sweeper drains them. You see the result as drive usage that your bucket totals do not explain.

This page names each such path, describes what you see inside it, states which sweeper is responsible, and gives the remediation.

The distinction determines both how you measure a path and whether you may ever delete from it by hand.

Per-drive paths hold raw files that one node writes directly to one drive. There is no erasure coding and no cross-node copy. Size them with du on one drive. Multiplying by the drive count estimates the deployment total, which holds only while the drives carry comparable load. Sum du across the drives when you need an exact figure. tmp, .trash, tmp-old, tmp/goroutines, multipart, .quarantine, logs/api, and logs/error are per-drive.

Cluster paths hold erasure-coded objects in the .minio.sys namespace, spread across every drive in the erasure set exactly like user objects. One logical entry occupies a shard on each drive. Deleting one drive’s shard does not remove the entry; it creates a heal target. buckets/*, config/*, batch-jobs/*, catalog/*, delta-sharing/*, and logs/audit are cluster paths.

Take the per-drive total on one node first. Growth in a per-drive path is usually node-local, so comparing two nodes tells you whether you are looking at a cluster-wide condition or a single stuck node.

du -sh -- /mnt/drive1/minio/.minio.sys/* /mnt/drive1/minio/.minio.sys/.[!.]*
find /mnt/drive1/minio/.minio.sys/tmp -mindepth 1 -maxdepth 1 -type d | wc -l
find /mnt/drive1/minio/.minio.sys/multipart -mindepth 2 -maxdepth 2 -type d | wc -l
find /mnt/drive1/minio/.minio.sys/.trash -mindepth 2 -maxdepth 2 | wc -l

Name the dot-directories explicitly, as above. A bare * glob skips .trash and .quarantine, which are two of the paths most likely to be holding the capacity you are looking for.

Then confirm from metrics whether the responsible sweeper is running at all. Read the counters as deltas over time: a counter that never advances is the signal, and its absolute value is not. Read the two gauges the opposite way. Compare minio_api_requests_stale_multipart_last_cleanup_seconds against api stale_uploads_cleanup_interval, and minio_system_drive_trash_last_cleanup_seconds against api delete_cleanup_interval. A value well past its interval means that sweeper has stalled.

Scope differs too. Only the trash metrics carry a drive label, so only those attribute to a single drive. The stale-multipart, stale-tmp, and quarantine metrics report per node, aggregated across that node’s drives.

What it is. Write staging. Every object write, multipart part upload, heal, and rename writes its erasure shards here first, then renames them into their final location on commit.

What you see. One directory per hour, named for the Unix second at which that hour began — 1786568400, for example. Each holds the writes that started during that hour: object writes one level deep under the first character of a UUID, multipart part directories named <uuid>x<unixnano>, and heal and rename staging named with a UUID. goroutines/ and the short-lived .writable-check-*.tmp files from the drive health checker sit at the top level, outside the hourly directories.

When it grows. A write that fails or is abandoned after the shards land but before the rename stays in its hourly directory until that directory ages out. A crash leaves everything that was in flight. Sustained growth means either a high rate of aborted writes or a stalled sweeper.

What reclaims it. Every api stale_uploads_cleanup_interval (default 6h), the sweep reads the top level of tmp and renames each expired entry into .trash/, where the trash purger removes it. An hourly directory expires one hour after api stale_uploads_expiry (default 24h): the directory keeps taking writes until its hour closes, so its youngest write is owed the full expiry. Entries with no hour in their name, such as goroutines/, are aged by modification time instead. Separately, the server renames the whole tmp directory to tmp-old/<uuid> at startup and deletes it in the background.

How to remediate. Restart the node — that moves the staging directory aside to tmp-old/<uuid> and deletes it in the background, so the space returns as that deletion progresses rather than at once. If you cannot restart, shorten api stale_uploads_expiry and api stale_uploads_cleanup_interval:

mc admin config set ALIAS api stale_uploads_expiry=6h stale_uploads_cleanup_interval=1h

Set the expiry above the longest single part you expect a client to stream. The expiry counts from when a write started, not from when it last wrote, so a part still streaming past the expiry has its directory renamed out from under it. That upload fails; no committed object is affected.

Deleting the contents by hand is safe with the server stopped on that node.

What it is. The deferred-delete staging area, a sibling of tmp rather than a directory inside it. Deletes rename the object directory, data directory, or xl.meta here rather than unlinking it on the request path, and a background purger removes it afterward. The tmp and multipart sweeps also hand their expired entries here, so one throttled purger does all the removal.

What you see. A short random prefix directory, and one entry per deferred delete inside it, named with a UUID. An entry is a file or a whole directory tree, depending on what was deleted. The prefix rotates every 100,000 deletes, which keeps any one directory from growing without bound.

When it grows. Any of three causes. The delete rate outruns the purger. The purger blocks on slow drive I/O. Or the pool is suspended for decommissioning, because a suspended pool skips immediate purges.

What reclaims it. The purger reads the top level of this directory every api delete_cleanup_interval (default 5m) and deletes every prefix directory it finds, pacing itself between deletes. It applies no age threshold: an entry is removed as soon as the purger sees it.

How to remediate. Confirm minio_system_drive_trash_purged_bytes_total is advancing for the affected drive. Read minio_system_drive_trash_purged_objects_total alongside it, but note that it counts prefix directories rather than individual deletes, so it advances far more slowly than the delete rate. If bytes are being purged and the directory is still growing, the workload’s delete rate is the cause and the backlog drains once it subsides. If nothing advances, check the drive for I/O errors and timeouts. Shortening the interval makes the purger run more often:

mc admin config set ALIAS api delete_cleanup_interval=1m

Deleting the contents by hand is safe with the server stopped on that node.

What it is. The previous run’s tmp directory, set aside at startup so the current run starts with an empty staging area.

What you see. One directory per server start, named with a UUID, containing whatever was in tmp when the previous run ended.

When it grows. The startup delete is asynchronous. A node that restarts repeatedly, or restarts while the delete is still running, accumulates one directory per restart.

How to remediate. It empties on its own once the node stays up. Deleting the contents by hand is safe with the server stopped on that node.

What it is. Post-incident profile dumps, written when the goroutine count or heap in use rises sharply against the day’s baseline.

What you see. One directory per UTC day of the month, named 01 through 31. Inside each, pairs of files named <YYYY-MM-DD>_<HHMMSS>Z-goroutines.txt.zst and <YYYY-MM-DD>_<HHMMSS>Z-mem.pprof.

When it grows. At most 100 dumps per day per drive. When the day rolls over, that day-of-month directory is deleted and recreated, so the content is at most one month old and does not survive a restart.

How to remediate. Nothing to do. Retrieve the dumps for a support case with mc support inspect ALIAS/.minio.sys/tmp/goroutines/**.

What it is. In-progress multipart uploads. Parts live here until CompleteMultipartUpload moves them into the bucket path.

What you see. Two levels: sha256(bucket/object)/<uploadID>/. The first level is the hex SHA-256 of the bucket and object name joined by /. The second is the upload ID, which is base64url(<UUID>x<UnixNano>) — the creation time is encoded in the name. Inside are xl.meta and one part.N shard per uploaded part.

When it grows. Clients that call CreateMultipartUpload and then neither complete nor abort the upload. This is the most common cause of unexplained .minio.sys growth, and it grows by the full size of every uploaded part.

What reclaims it. Every api stale_uploads_cleanup_interval (default 6h), upload directories older than api stale_uploads_expiry (default 24h) are moved to .trash/, and empty hash directories older than the same expiry are removed.

How to remediate. Count the upload directories per drive with the find command above and compare against what clients report as in flight. Shorten the expiry so abandoned uploads are reclaimed sooner:

mc admin config set ALIAS api stale_uploads_expiry=6h

For per-bucket control, add an AbortIncompleteMultipartUpload lifecycle rule, which is enforced independently of this server-wide sweep.

What it is. Forensic copies of dangling objects — objects whose remaining xl.meta copies fell below what is needed to reconstruct them. Populated only when heal dangling is set to quarantine.

What you see. <bucket>/<object-path>/<UnixNano>/, where <UnixNano> is the nanosecond timestamp at which the object was quarantined. Inside are the xl.meta and, when the data directory still existed, a <data-dir-uuid>/ holding the part.N shards. A stats file appears at the top level while healing traces are subscribed.

When it grows. Every dangling object detected while heal dangling=quarantine adds an entry, and the default retention is one year. On a deployment with recurring dangling objects this accumulates for that full year before anything is reclaimed.

What reclaims it. The first sweep runs 5 to 10 minutes after the node starts, and one runs every 24 hours per erasure set afterward. Each deletes entries whose timestamp is older than heal dangling_expiry (default 1y). Nothing is deleted when heal dangling is not quarantine.

How to remediate. Shorten the retention, or stop quarantining and purge dangling objects outright:

mc admin config set ALIAS heal dangling_expiry=7d
mc admin config set ALIAS heal dangling=purge

See heal dangling and heal dangling_expiry.

What it is. Recorded API, error, and audit logs, when log recording is enabled.

What you see. logs/api/ and logs/error/ hold raw zstd-compressed files on the drive. logs/audit/ holds erasure-coded objects in the .minio.sys namespace, not raw files — an active audit.log plus rotated <timestamp>.log.zst objects.

When it grows. The API and error recorders are bounded by their drive_limit setting, applied per recorder per drive. Audit logs are not covered by drive_limit and are accounted for separately; the active object rotates at 1 GiB and rotated objects are not expired.

How to remediate. See where recorded logs are stored for the full layout, the rotation behavior, and the settings that bound each recorder.

What it is. Cached listing results. A ListObjects walk that spans more than one response writes its results here so that follow-up pages resume instead of re-walking.

What you see. One directory per listing, named with the listing ID, containing block-0.s2, block-1.s2, and so on — compressed batches of listing entries.

When it grows. Heavy listing traffic, especially many concurrent listings that clients abandon after the first page.

What reclaims it. A reaper runs every minute and drops caches that are not worth keeping: an unfinished listing with no update for one minute, a finished listing 15 minutes after the last client contact, and a failed listing after five minutes. A bucket keeps at most 5000 caches; the rest are removed oldest first. At startup, every bucket’s .metacache directory is renamed into the trash.

How to remediate. Restart the affected node to clear it. Sustained growth points at the listing workload — reduce concurrent full-bucket listings, or have clients page through to completion.

This is the one cluster path where removing a single drive’s copy is safe, because the server does exactly that itself at startup. With the server stopped on a node, you may delete that node’s .metacache directories. A missing or incomplete cache makes the listing recompute; it does not leave an object short of its parity.

What it is. Delete-replication trackers. Deleting a versioned object in a bucket that uses purge-on-delete writes the state of its whole version stack here first. The tracker stays until two things finish: every version replicates, and any tiered content leaves the remote tier.

What you see. Two levels of hex: <first 3 hex>/<remaining 61 hex>.bin, where the hex is the SHA-256 of the bucket and object name. Each tracker is itself a versioned object, so one path can carry several versions.

When it grows. Trackers are only deleted once replication and tier cleanup complete. A replication target that is unreachable, misconfigured, or removed while purge-on-delete is still enabled leaves every tracker in place, and each subsequent delete adds another.

How to remediate. Fix the replication target so the backlog drains. Check the bucket’s replication targets and its replication backlog first; the trackers are a consequence of stalled replication, not an independent problem. Remove the purge-on-delete configuration if the target is gone for good.

What it is. Markers for buckets that were deleted but whose deletion has not yet been confirmed across every site in a site-replication configuration.

What you see. One entry per deleted bucket, named after the bucket.

When it grows. A site that is unreachable, or a site-replication configuration that never reaches agreement, leaves markers in place. Each entry is a directory stub and consumes essentially no capacity.

How to remediate. Restore connectivity to the peer sites. The markers are purged once all sites confirm the deletion.

What it is. Snapshots of the server configuration, one per configuration change, kept so a change can be rolled back.

What you see. One object per snapshot, named <uuid>.kv.

When it grows. One entry per mc admin config set. Nothing expires them, so automation that rewrites configuration on a schedule adds an entry every time it runs.

How to remediate. List the history, then clear it if you do not need the restore points:

mc admin config history ALIAS
mc admin config history ALIAS --clear

Clearing does not change the current configuration, but it removes the restore IDs that mc admin config restore depends on. Each entry is small, so treat this as housekeeping rather than a capacity problem unless configuration is being rewritten programmatically.

What it is. Records for temporary credentials issued through STS — AssumeRole, AssumeRoleWithWebIdentity, LDAP, and certificate-based logins.

What you see. sts/<access-key>/identity.json holding the credential, and policydb/sts-users/<access-key>.json holding its policy mapping. Alongside them, config/iam/token-revokes/ holds one entry per revoked session token, bounded by how often you revoke.

When it grows. One pair of entries per credential issued. The periodic IAM refresh scans the STS prefix and drops expired records. A high issuance rate combined with a long credential lifetime leaves a large working set behind. The scan then slows as the count rises, and the server logs IAM expired STS purge took Ns when a refresh runs long. Treat that log line as the early warning.

How to remediate. Shorten credential lifetime at the source: reduce the requested DurationSeconds on AssumeRole calls, or lower the server-side STS duration in the STS settings. Have applications reuse a valid credential until it nears expiry rather than assuming a role per request. Watch for the purge-duration log line as the early warning.

What it is. Batch job definitions and their progress reports.

What you see. batch-jobs/<job-id> holding the submitted job definition, and batch-jobs/reports/<job-id>/batch-<type>.bin holding its report, where <type> is replicate, rotate, expire, or catalog.

When it grows. One pair of entries per submitted job. A cleanup pass runs every 12 hours. It removes a job and its report once the job has completed or failed and has gone three days without an update, or 60 days for catalog jobs. It also removes a report whose job definition has already gone. A report written before 2025 uses an older ID format that the pass cannot match to a job, so it is removed once it is 60 days old. A job that never reaches a terminal state keeps both its definition and its report indefinitely.

How to remediate. Check for jobs stuck in a non-terminal state and cancel them so the cleanup pass can reclaim them. Automation that submits jobs on a short schedule produces entries faster than the three-day window releases them; reduce the submission rate or consolidate jobs.

What it is. AIStor Tables catalog metadata — warehouse and namespace registries, table metadata pointers, and in-flight transaction logs.

What you see. catalog/<warehouse>/warehouse.bin and namespace.bin registries, catalog/<warehouse>/tables/<uuid>.bin metadata pointers, staged metadata under a hierarchical path derived from the table UUID, and catalog/<warehouse>/.tx/pending/<transaction-id>.bin transaction logs.

When it grows. The .tx/pending/ directory is the transient part. A commit writes a log there and removes it when the commit finishes. A commit interrupted by a crash or a node failure leaves its log behind for recovery to replay. Logs that accumulate mean recovery is not completing.

How to remediate. Count the pending logs for the affected warehouse. A handful that clear within minutes is normal operation. A count that only rises indicates catalog recovery is failing — collect the server logs for the warehouse and open a SUBNET issue.

What it is. Delta Sharing configuration, shares, and bearer tokens.

What you see. delta-sharing/config.json, delta-sharing/shares/ with one entry per share, and delta-sharing/tokens/ with one entry per issued token.

When it grows. One entry per token issued. A token created without an explicit expiry never expires, so its record is never reclaimed. Nothing removes a token record other than expiry or an explicit removal.

How to remediate. Remove the tokens you no longer need, and give new ones an explicit expiry so they age out on their own:

mc table share token list ALIAS/SHARE
mc table share token remove ALIAS TOKEN_ID
mc table share token create ALIAS/SHARE --expires 30d

See mc table share token.

These appear alongside the paths above and are frequently mistaken for workload growth. Most entries are a single small file with a fixed name. The rest are bounded collections: the three newest heal tracker backups, one usage-cache file per erasure set, one resync file per peer site, and one file per bucket configuration. None of them grow with object count or request volume.

Reach for the filesystem only after the responsible sweeper has been ruled out, and only within these limits.

Safe to delete, with the AIStor server stopped on that node:

• .minio.sys/tmp/ contents
• .minio.sys/tmp-old/ contents
• .minio.sys/.trash/ contents
• .minio.sys/buckets/<bucket>/.metacache/ contents
• .minio.sys/.quarantine/ contents, accepting the loss of the forensic copies

Never delete by hand:

• .minio.sys/multipart/ — aborts live uploads
• .minio.sys/config/ — the server configuration and the IAM database
• .minio.sys/catalog/ — the authoritative table catalog
• .minio.sys/buckets/.trackdelete/ — orphans content on replication targets and remote tiers
• .minio.sys/buckets/<bucket>/ configuration files — the bucket’s settings
• .minio.sys/format.bin — the drive’s erasure set identity

Open a SUBNET issue if any of these hold:

• A sweeper’s metric is flat while its path keeps growing, and the drive reports no I/O errors.
• The count under catalog/<warehouse>/.tx/pending/ only rises.
• A path listed as fixed above is not fixed in your deployment.

Collect these first:

• du output per .minio.sys subdirectory from an affected drive.
• The counters for that path from the table above.
• mc admin config get ALIAS api and mc admin config get ALIAS heal.