Healing

Healing is MinIO AIStor’s ability to restore an object that has been damaged, corrupted, or partially lost. Damage or loss can come from several sources, including:

• drive-level errors or failure
• OS or filesystem errors or failure
• bit rot

Whether MinIO AIStor can restore a damaged object depends on three variables:

• Total drives in the erasure set the object is part of

• Available drives that contain intact parts of the object

• Parity settings for the erasure set

Parity specifies how many recovery shards MinIO AIStor creates when it writes an object. The number of data shards is the erasure set size minus the parity. MinIO AIStor distributes shards randomly across the drives in the erasure set, so that no single drive holds only parity shards or only data shards.

MinIO AIStor can use either data or parity shards to reconstruct an object, as long as the number of intact shards is at least the erasure set’s data shard count. MinIO AIStor can then heal the missing data or parity shards while returning the object to the calling client.

Use mc admin object info to summarize the current state of an object on disk. This command outputs a summary of all of the shards of an object (also called “parts”), including any that are missing or damaged.

MinIO AIStor automatically checks the consistency of an object’s data shards each time you request an object with a GET or HEAD operation. For versioned buckets, MinIO AIStor also checks for consistency during PUT operations.

If all of the data shards are found intact, MinIO AIStor serves the object from the data shards without inspecting the corresponding parity shards.

If the object has missing or damaged data shards, MinIO AIStor uses the available parity shards to heal the object before serving it as part of the operation. MinIO AIStor must have at least as many intact shards, of either type, as the erasure set has data shards. Otherwise, MinIO AIStor cannot recover the object. If a parity shard is lost or damaged, MinIO AIStor restores it, provided enough other parity shards remain to serve the object.

MinIO AIStor uses an object scanner to perform several object-related tasks. One of these tasks checks object integrity and heals any object it finds damaged or corrupted.

On each scanning pass, MinIO AIStor uses a hash of the object name to select one out of every 1,024 objects to check.

If any object is found to have lost shards, MinIO AIStor heals the object from available shards. By default, MinIO AIStor does not check for bit rot corruption using the scanner. This check is expensive to perform, and the risk of bit rot across multiple drives is low.

Consult with MinIO Engineers before manually starting a healing process on a deployment.

When a drive goes offline, MinIO AIStor tracks the duration the drive is offline. If a drive is offline for more than 48 hours, MinIO AIStor also determines whether the erasure set was also offline for the same period.

If both the erasure set and drive were offline for the same period, MinIO AIStor performs no healing. This is because no PUT or DELETE requests would have been possible during the offline time.

If the erasure set remained available for write operations and only the drive was offline, MinIO AIStor heals the objects on the drive. If the drive was offline for more than 48 hours, MinIO AIStor treats the drive as if it were a fresh drive and heals all objects.

MinIO AIStor publishes healing metrics under the /cluster/erasure-set endpoint. Use them to monitor the status of healing processes on a deployment.

Bit rot is silent data corruption from random changes at the storage media level. For data drives, it is typically the result of decay of the electrical charge or magnetic orientation that represents the data. These sources can range from the small current spike during a power outage to a random cosmic ray resulting in flipped bits. The resulting “bit rot” can cause subtle errors or corruption on the data medium without triggering monitoring tools or hardware.

MinIO AIStor’s optimized implementation of the HighwayHash algorithm captures and heals corrupted objects on the fly. Integrity is ensured from end to end by computing a hash on READ and verifying it on WRITE from the application, across the network, and to the memory or drive. The implementation is designed for speed and can achieve hashing speeds over 10 GB/sec on a single core on Intel CPUs.

You can complete a bit rot scan on a specific object by running mc admin object info with the --bitrot flag.