Scale MinIO KMS on Linux

MinIO KMS is a distributed keystore and supports scaling through adding or removing MinIO KMS nodes to an existing cluster. You can add or remove nodes from a MinIO KMS cluster on demand.

Keep in mind the cluster fault tolerance considerations.

Add a MinIO KMS node

Retrieve the HSM key used on the existing node(s) of the cluster.

For example, retrieve the key from a secrets manager.
Install MinIO KMS on the new node using the same HSM key as the other cluster nodes.

See the Linux installation guide for details.

You must use the same HSM key on each node of the cluster to allow for inter-node communication.

Be sure to start the MinIO KMS process.
From an existing node on the cluster, add the new node to the cluster.
```
minkms add https://minkms4.example.net:7373 --api-key ROOT-API-KEY-STRING
```
The API key can be retrieved from the MinIO KMS startup logs or computed from the HSM with
```
minkms identity hsm:aes256:KEYVALUE
```
Do not run minkms add from the new node. Run the command from a node already in the MinIO KMS cluster.
Update any load balancers, reverse proxies, or other network control components to include the new host as part of the configuration.

Remove a MinIO KMS node

To perform write operations, all nodes on the cluster must be available.

Healthy cluster

A healthy cluster is one where all nodes are available and online, with full connectivity between them.

The following command removes a node hosted at https://minkms5.example.net:7373 from a healthy MinIO KMS cluster with complete connectivity to all nodes. Run the command from any of the nodes that will remain in the cluster.

minkms rm https://minkms5.example.net:7373 --api-key ROOT-API-KEY-STRING

The API key can be retrieved from the MinIO KMS startup logs or computed from the HSM.

minkms identity hsm:aes256:KEYVALUE

Unhealthy cluster

An unhealthy cluster is one where one ore more of the nodes are inaccessible from the rest of the cluster and no write requests can be processed.

If a node is temporarily or permanently lost, run minkms edit --rm to remove the node from the cluster configuration. After removing lost node(s), the remaining nodes resume write functionality.

Complete the following on a live node on the cluster that will remain part of the cluster.

Retrieve the ID for the unhealthy node.
```
minkms ls
```
Edit the cluster configuration to remove the unhealthy node.
```
minkms edit --rm NODE-ID --api-key ROOT-API-KEY-STRING
```
The API key can be retrieved from the MinIO KMS startup logs or computed from the HSM.
```
minkms identity hsm:aes256:KEYVALUE
```

Repeat the process for any other lost nodes.