Decommission AIStor Server Pool on Linux

The following procedures decommissions one or more pools on a Linux baremetal deployment.

Review the AIStor deployment topology

The mc admin decommission command returns a list of all pools in the AIStor deployment:

mc admin decommission status myaistor

The command returns output similar to the following:

┌─────┬──────────────────────────────────────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                                                            │ Capacity                         │ Status │
│ 1st │ https://aistor-{01...04}.example.com:9000/mnt/disk{1...4}/aistor │  10 TiB (used) / 10  TiB (total) │ Active │
│ 2nd │ https://aistor-{05...08}.example.com:9000/mnt/disk{1...4}/aistor │  60 TiB (used) / 100 TiB (total) │ Active │
│ 3rd │ https://aistor-{09...12}.example.com:9000/mnt/disk{1...4}/aistor │  40 TiB (used) / 100 TiB (total) │ Active │
└─────┴──────────────────────────────────────────────────────────────────┴──────────────────────────────────┴────────┘

The example deployment above has three pools. Each pool has four servers with four drives each.

Identify the target pool(s) for decommissioning and review the current capacity. The remaining pools in the deployment must have sufficient total capacity to migrate all object stored in the decommissioned pool.

Update network control plane

Update any load balancers, reverse proxies, or other network control planes to remove the decommissioned server pool from the connection configuration for the AIStor deployment. The remaining pools automatically route operations as-needed to the pool during the decomissioning process.
Start the decommissioning process

Use the mc admin decommission start command to begin decommissioning the target pool. Specify the alias of the deployment and the full description of the pool to decommission, including all hosts, disks, and file paths.

For example, given the previous output the following command would decommission the first pool in the deployment
```
mc admin decommission start myaistor/ https://aistor-{01...04}.example.net:9000/mnt/disk{1...4}/aistor
```
You can specify multiple pools to the command if decommissioning a larger scope of hardware.

During the decommissioning process, AIStor continues routing read operations (GET, LIST, HEAD) to the pool for those objects not yet migrated. AIStor routes all new write operations (PUT) to the remaining pools in the deployment.

Load balancers, reverse proxy, or other network control components which manage connections to the deployment do not need to modify their configurations at this time.

Monitor the decommissioning process

Use the mc admin decommission status command to monitor the decommissioning process.

mc admin decommission status myaistor

The command returns output similar to the following:

┌─────┬──────────────────────────────────────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                                                            │ Capacity                         │ Status   │
│ 1st │ https://aistor-{01...04}.example.com:9000/mnt/disk{1...4}/aistor │  10 TiB (used) / 10  TiB (total) │ Draining │
│ 2nd │ https://aistor-{05...08}.example.com:9000/mnt/disk{1...4}/aistor │  60 TiB (used) / 100 TiB (total) │ Active   │
│ 3rd │ https://aistor-{09...12}.example.com:9000/mnt/disk{1...4}/aistor │  40 TiB (used) / 100 TiB (total) │ Active   │
└─────┴──────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────┘

You can retrieve more detailed information by specifying the description of the server pool to the command:

mc admin decommission status myaistor https://aistor-{01...04}.example.com:9000/mnt/disk{1...4}/aistor

The command returns output similar to the following:

Decommissioning rate at 100MiB/sec [1TiB/10TiB]
Started: 30 minutes ago

mc admin decommission status marks the Status as Complete once decommissioning is completed. You can move on to the next step once decommissioning is completed.

If Status reads as failed, you can re-run the mc admin decommission start command to resume the process. For persistent failures, use mc admin logs or review the systemd logs (e.g. journalctl -u aistor) to identify more specific errors.

Remove the decommissioned pool from the deployment configuration

Open the environment file /etc/default/minio and modify the MINIO_VOLUMES to remove the decommissioned pool

For example, given the example output the MINIO_VOLUMES variable would resemble the following:
```
MINIO_VOLUMES="https://aistor-{1...4}.example.net:9000/mnt/disk{1...4}/aistor https://aistor-{5...8}.example.net:9000/mnt/disk{1...4}/aistor https://aistor-{9...12}.example.net:9000/mnt/disk{1...4}/aistor"
```
In the above example, the new value should instead show:
```
MINIO_VOLUMES="https://aistor-{5...8}.example.net:9000/mnt/disk{1...4}/aistor https://aistor-{9...12}.example.net:9000/mnt/disk{1...4}/aistor"
```
Remove all pools that have successfully completed decommissioning.

Ensure all nodes have identical environment files before restarting the cluster using mc admin service restart.
Restart the AIStor deployment

Issue the following commands on each node simultaneously in the deployment to restart the AIStor service:
```
sudo systemctl restart aistor.service
```
Use the following commands to confirm the service is online and functional:
```
sudo systemctl status aistor.service
journalctl -f -u aistor.service
```
AIStor may log an increased number of non-critical warnings while the server processes connect and synchronize. These warnings are typically transient and should resolve as the deployment comes online.

MinIO strongly recommends restarting all Object Store processes in a deployment simultaneously. AIStor operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (that is, one node at a time) restarts.

Once the deployment is online, use mc admin info to confirm the uptime of all remaining servers in the deployment.