Recover after drive failure
AIStor supports hot-swapping failed drives with new healthy drives. AIStor detects and heals those drives without requiring any node or deployment-level restart. AIStor healing occurs only on the replaced drive(s) and in most cases has minimal or negligible impact on deployment performance.
AIStor healing ensures consistency and correctness of all data restored onto the drive.
If multiple drives within the same erasure set require healing, AIStor prioritizes the first drive that begins healing within the erasure set. Other drives in the erasure set wait until healing finishes on the first drive before the next drive starts healing.
This allows drives that begin healing to complete the process as quickly as possible.
The following steps provide a more detailed walkthrough of drive replacement.
These steps assume an AIStor deployment where each node manages drives using /etc/fstab with per-drive labels.
1) Comment out the failed drive(s)
Open /etc/fstab in your preferred text editor and comment out the drives identified as failed or failing by placing a # at the beginning of the line.
For example, the following configuration comments out two failed drives
#/etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
#LABEL=DRIVE1 /mnt/drive1 xfs defaults,noatime 0 2
LABEL=DRIVE2 /mnt/drive2 xfs defaults,noatime 0 2
LABEL=DRIVE3 /mnt/drive3 xfs defaults,noatime 0 2
LABEL=DRIVE4 /mnt/drive4 xfs defaults,noatime 0 2
LABEL=DRIVE5 /mnt/drive5 xfs defaults,noatime 0 2
LABEL=DRIVE6 /mnt/drive6 xfs defaults,noatime 0 2
LABEL=DRIVE7 /mnt/drive7 xfs defaults,noatime 0 2
#LABEL=DRIVE8 /mnt/drive8 xfs defaults,noatime 0 2
Follow your operating system guidance for any additional commands required after updating /etc/fstab.
Commenting the drive ensures that the node can start correctly in the event of a system restart.
Do not use nofail or similar methods to force the OS to start with failing drives.
2) Unmount the failed drive(s)
Unmount each failed drive using umount.
The following command assumes a mount path of /dev/nvme1 and /dev/nvme9 for the two failed drives in the previous example:
umount /dev/nvme1
umount /dev/nvme9
3) Replace the failed drive(s)
Remove the failed drive(s) from the node hardware and replace it with known healthy drive(s). Replacement drives must meet the following requirements:
- XFS formatted and empty.
- Same drive type (for example, HDD, SSD, NVMe).
- Equal or greater performance.
- Equal or greater capacity.
Using a replacement drive with greater capacity does not increase the total cluster storage. AIStor uses the smallest drive’s capacity as the ceiling for all drives in the server pool.
The following example command formats a drive as XFS and assigns it a label to match the failed drives.
mkfs.xfs /dev/nvme1 -L DRIVE1
mkfs.xfs /dev/nvme9 -L DRIVE8
Use label-based mounting to ensure consistent drive order that persists through system restarts.
4) Update fstab
Open /etc/fstab in your preferred text editor and uncomment the lines associated to the replaced drives:
$ cat /etc/fstab
LABEL=DRIVE1 /mnt/drive1 xfs defaults,noatime 0 2
LABEL=DRIVE2 /mnt/drive2 xfs defaults,noatime 0 2
LABEL=DRIVE3 /mnt/drive3 xfs defaults,noatime 0 2
LABEL=DRIVE4 /mnt/drive4 xfs defaults,noatime 0 2
LABEL=DRIVE5 /mnt/drive5 xfs defaults,noatime 0 2
LABEL=DRIVE6 /mnt/drive6 xfs defaults,noatime 0 2
LABEL=DRIVE7 /mnt/drive7 xfs defaults,noatime 0 2
LABEL=DRIVE8 /mnt/drive8 xfs defaults,noatime 0 2
If you changed the label used or are using UUID-based mounting, update the label or UUID reference to correspond with the new mount configuration.
Follow your operating system guidance for any additional commands required after updating /etc/fstab.
Cloud environment instances which depend on mounted external storage may encounter boot failure if one or more of the remote file mounts return errors or failure.
For example, an AWS ECS instances with mounted persistent EBS volumes may fail to boot with the standard /etc/fstab configuration if one or more EBS volumes fail to mount.
You can set the nofail option to silence error reporting at boot and allow the instance to boot with one or more mount issues.
Do not use nofail outside of cloud environments which showcase the aformentioned behavior.
Silencing drive errors prevents both AIStor and the OS from responding to those errors in a normal fashion.
5) Remount the replaced drive(s)
Use mount -a to remount the drives unmounted at the beginning of this procedure:
mount -a
The command should result in remounting of all of the replaced drives.
Use df -h or lsblk to verify the result of the mounting operation.
6) Monitor AIStor for drive detection and healing status
Use mc admin logs command or journalctl -u minio for systemd-managed installations to monitor the server log output after remounting drives.
The output should include messages identifying each formatted and empty drive.
Use mc admin heal to monitor the overall healing status on the deployment.
AIStor aggressively heals replaced drive(s) to ensure rapid recovery from the degraded state.
Next steps
- Monitor the cluster for any further drive failures.
- Some drive batches may fail in close proximity to each other.
- Deployments seeing higher than expected drive failure rates should schedule dedicated maintenance around replacing the known bad batch.
- Consider using SUBNET to coordinate with MinIO engineering around guidance for any such operations.