Decommission a Server Pool

Networking and Firewalls

Each node should have full bidirectional network access to every other node in the deployment. For containerized or orchestrated infrastructures, this may require specific configuration of networking and routing components such as ingress or load balancers. Certain operating systems may also require setting firewall rules. For example, the following command explicitly opens the default MinIO server API port 9000 on servers using firewalld:

firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --reload

If you set a static MinIO Console port (e.g. :9001) you must also grant access to that port to ensure connectivity from external clients.

MinIO strongly recomends using a load balancer to manage connectivity to the cluster. The Load Balancer should use a “Least Connections” algorithm for routing requests to the MinIO deployment, since any MinIO node in the deployment can receive, route, or process client requests.

The following load balancers are known to work well with MinIO:

• NGINX

• HAProxy

Configuring firewalls or load balancers to support MinIO is out of scope for this procedure.

Deployment Must Have Sufficient Storage

The decommissioning process migrates objects from the target pool to other pools in the deployment. The total available storage on the deployment must exceed the total storage of the decommissioned pool.

For example, consider a deployment with the following distribution of used and free storage:

Decommissioning Pool 1 requires distributing the 100TB of used storage across the remaining pools. Pool 2 and Pool 3 each have 100TB of unused storage space and can safely absorb the data stored on Pool 1.

However, if Pool 1 were full (e.g. 200TB of used space), decommissioning would completely fill the remaining pools and potentially prevent any further write operations.

Decommissioning Does Not Support Tiering

MinIO does not support decommissioning pools in deployments with tiering configured. The MinIO server rejects decommissioning attempts if any bucket in the deployment has a tiering configuration.

Decommissioning Ignores Delete Markers

MinIO does not migrate objects whose only remaining version is a delete markers. This avoids creating empty metadata on the remaining server pools for objects already considered fully deleted.

Decommissioning is Resumable

MinIO resumes decommissioning if interrupted by transient issues such as deployment restarts or network failures.

For manually cancelled or failed decommissioning attempts, MinIO resumes only after you manually re-initiate the decommissioning operation.

The pool remains in the decommissioning state regardless of the interruption. A pool can never return to active status after decommissioning begins.

Decommissioning is Non-Disruptive

Removing a decommissioned server pool requires restarting all MinIO nodes in the deployment at around the same time.

MinIO strongly recommends restarting all nodes simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

1) Review the MinIO Deployment Topology

The mc admin decommission command returns a list of all pools in the MinIO deployment:

mc admin decommission status myminio

The command returns output similar to the following:

┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                                                          │ Capacity                         │ Status │
│ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │  10 TiB (used) / 10  TiB (total) │ Active │
│ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │  60 TiB (used) / 100 TiB (total) │ Active │
│ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │  40 TiB (used) / 100 TiB (total) │ Active │
└─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴────────┘

The example deployment above has three pools. Each pool has four servers with four drives each.

Identify the target pool for decommissioning and review the current capacity. The remaining pools in the deployment must have sufficient total capacity to migrate all object stored in the decommissioned pool.

In the example above, the deployment has 210TiB total storage with 110TiB used. The first pool (minio-{01...04}) is the decommissioning target, as it was provisioned when the MinIO deployment was created and is completely full. The remaining newer pools can absorb all objects stored on the first pool without significantly impacting total available storage.

2) Start the Decommissioning Process

Use the mc admin decommission start command to begin decommissioning the target pool. Specify the alias of the deployment and the full description of the pool to decommission, including all hosts, disks, and file paths.

mc admin decommission start myminio/ https://minio-{01...04}.example.net:9000/mnt/disk{1...4}/minio

The example command begins decommissioning the matching server pool on the myminio deployment.

During the decommissioning process, MinIO continues routing read operations (GET, LIST, HEAD) operations to the pool for those objects not yet migrated. MinIO routes all new write operations (PUT) to the remaining pools in the deployment.

Load balancers, reverse proxy, or other network control components which manage connections to the deployment do not need to modify their configurations at this time.

3) Monitor the Decommissioning Process

Use the mc admin decommission status command to monitor the decommissioning process.

mc admin decommission status myminio

The command returns output similar to the following:

┌─────┬────────────────────────────────────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                                                          │ Capacity                         │ Status   │
│ 1st │ https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio │  10 TiB (used) / 10  TiB (total) │ Draining │
│ 2nd │ https://minio-{05...08}.example.com:9000/mnt/disk{1...4}/minio │  60 TiB (used) / 100 TiB (total) │ Active   │
│ 3rd │ https://minio-{09...12}.example.com:9000/mnt/disk{1...4}/minio │  40 TiB (used) / 100 TiB (total) │ Active   │
└─────┴────────────────────────────────────────────────────────────────┴──────────────────────────────────┴──────────┘

You can retrieve more detailed information by specifying the description of the server pool to the command:

mc admin decommission status myminio https://minio-{01...04}.example.com:9000/mnt/disk{1...4}/minio

The command returns output similar to the following:

Decommissioning rate at 100MiB/sec [1TiB/10TiB]
Started: 30 minutes ago

mc admin decommission status marks the Status as Complete once decommissioning is completed. You can move on to the next step once decommissioning is completed.

If Status reads as failed, you can re-run the mc admin decommission start command to resume the process. For persistent failures, use mc admin console or review the systemd logs (e.g. journalctl -u minio) to identify more specific errors.

4) Remove the Decommissioned Pool from the Deployment Configuration

Once decommissioning completes, you can safely remove the pool from the deployment configuration. Modify the startup command for each remaining MinIO server in the deployment and remove the decommissioned pool.

The .deb or .rpm packages install a systemd service file to /etc/systemd/system/minio.service. For binary installations, this procedure assumes the file was created manually as per the Deploy MinIO in Distributed Mode procedure.

The minio.service file uses an environment file located at /etc/default/minio for sourcing configuration settings, including the startup. Specifically, the MINIO_VOLUMES variable sets the startup command:

cat /etc/default/minio | grep "MINIO_VOLUMES"

The command returns output similar to the following:

MINIO_VOLUMES="https://minio-{1...4}.example.net:9000/mnt/disk{1...4}/minio https://minio-{5...8}.example.net:9000/mnt/disk{1...4}/minio https://minio-{9...12}.example.net:9000/mnt/disk{1...4}/minio"

Edit the environment file and remove the decommissioned pool from the MINIO_VOLUMES value.

5) Update Network Control Plane

Update any load balancers, reverse proxies, or other network control planes to remove the decommissioned server pool from the connection configuration for the MinIO deployment.

Specific instructions for configuring network control plane components is out of scope for this procedure.

6) Restart the MinIO Deployment

Issue the following commands on each node simultaneously in the deployment to restart the MinIO service:

sudo systemctl restart minio.service

Use the following commands to confirm the service is online and functional:

sudo systemctl status minio.service
journalctl -f -u minio.service

MinIO may log an increased number of non-critical warnings while the server processes connect and synchronize. These warnings are typically transient and should resolve as the deployment comes online.

MinIO strongly recommends restarting all nodes simultaneously. MinIO operations are atomic and strictly consistent. As such the restart procedure is non-disruptive to applications and ongoing operations.

Do not perform “rolling” (e.g. one node at a time) restarts.

Once the deployment is online, use mc admin info to confirm the uptime of all remaining servers in the deployment.