Data Recovery

Distributed AIStor deployments rely on Erasure Coding to provide built-in tolerance for multiple drive or node failures. Depending on the deployment topology and the selected erasure code parity, AIStor can tolerate the loss of up to half the drives or nodes in the deployment while maintaining read access (“read quorum”) to objects.

The following table lists the typical types of failure in an AIStor deployment and links to procedures for recovering from each:

Since AIStor can operate in a degraded state without significant performance loss, administrators can schedule hardware replacement in proportion to the rate of hardware failure. “Normal” failure rates (single drive or node failure) may allow for a more reasonable replacement timeframe, while “critical” failure rates (multiple drives or nodes) may require a faster response.

Sometimes nodes contain one or more drives that are either partially failed or operating in a degraded state. Symptoms of such drives may include increasing drive errors, SMART warnings, timeouts in AIStor logs, and so on. You can safely unmount an unhealthy drive if the cluster has sufficient remaining healthy drives to maintain read and write quorum. Missing drives are less disruptive to the deployment than drives that are consistently producing read and write errors.