Node Maintenance

AIStor allows you to temporarily remove nodes from active service for planned maintenance operations. Removing nodes allows administrators to gracefully take nodes offline without disrupting cluster operations, similar to the cordon functionality in Kubernetes. A cordoned node finishes in-flight operations and marks itself as unavailable for any other operation. Use this to do hardware maintenance on a node, complete operating system updates, or perform troubleshooting.

The following diagram illustrates the node state transitions during the maintenance workflow:

The mc admin cordon command removes a node from active service. By default, the command initiates a graceful drain of existing connections before fully cordoning the node.

Replace ALIAS with your AIStor cluster alias and NODE with the target node address (for example, node1.example.com:9000).

When you cordon a node, AIStor performs a graceful drain of existing connections:

• The node enters a draining state.
• The health endpoint returns HTTP 503, preventing new client requests from routing to this node.
• AIStor waits up to two minutes to allow existing connections to complete.

After draining completes, the node transitions to a fully cordoned state. AIStor disconnects all grid connections.

To cordon a node immediately, you can skip the drain phase using the --no-drain flag:

mc admin cordon --no-drain ALIAS NODE

Use mc admin info to view the status of nodes in your cluster, including cordoned and draining nodes:

mc admin info ALIAS

Nodes display one of the following states:

Use the mc admin uncordon command to direct the node to return to active service.

You must manually restart the AIStor process on a cordoned node before running mc admin uncordon, such as by running sudo systemctl restart minio on the node. The uncordon command does not restart the node.

AIStor persists the cordon state to storage. If a draining or cordoned node restarts before being uncordoned, it automatically re-enters the cordoned state. A draining node that restarts transitions directly to the fully cordoned state.

This behavior ensures that nodes do not accidentally rejoin the cluster during maintenance windows.

Before allowing a cordon or drain operation, AIStor validates that the operation does not cause the cluster to lose quorum. If cordoning the node would reduce the cluster below the minimum required nodes for read and write operations, the command fails with an error similar to the following:

cluster would lose quorum

For clusters operating near minimum quorum, verify the impact of taking a node offline before cordoning. Use mc admin info to review current cluster health and capacity.

When running AIStor on Kubernetes, the cordon workflow requires additional considerations for Pod lifecycle management.

The mc admin cordon command operates at the AIStor application layer, not the Kubernetes node layer. It removes an AIStor Pod from cluster participation while the Pod continues running. This differs from kubectl cordon, which prevents new Pods from scheduling on a Kubernetes node.

For AIStor maintenance, use mc admin cordon to gracefully stop activity on a Pod on the AIStor cluster before performing maintenance on the underlying infrastructure.