Scale Key Manager on Linux

Key Manager is a distributed keystore and supports scaling through adding or removing Key Manager nodes to an existing cluster. You can add or remove nodes from a Key Manager cluster on demand.

Keep in mind the cluster fault tolerance considerations.

Add a Key Manager node

Retrieve the HSM key used on the existing node(s) of the cluster.

For example, retrieve the key from a secrets manager.
Install Key Manager on the new node using the same HSM key as the other cluster nodes.

See the Linux installation guide for details.

You must use the same HSM key on each node of the cluster to allow for inter-node communication.

Be sure to start the Key Manager process.
From an existing node on the cluster, add the new node to the cluster.
```
minkms add https://keymanager4.example.net:7373 --api-key ROOT-API-KEY-STRING
```
The API key can be retrieved from the Key Manager startup logs or computed from the HSM with
```
minkms identity hsm:aes256:KEYVALUE
```
Do not run minkms add from the new node. Run the command from a node already in the Key Manager cluster.
Update any load balancers, reverse proxies, or other network control components to include the new host as part of the configuration.

Remove a Key Manager node

To perform write operations, all nodes on the cluster must be available.

Healthy cluster

A healthy cluster is one where all nodes are available and online, with full connectivity between them.

The following command removes a node hosted at https://keymanager5.example.net:7373 from a healthy Key Manager cluster with complete connectivity to all nodes. Run the command from any of the nodes that will remain in the cluster.

minkms rm https://keymanager5.example.net:7373 --api-key ROOT-API-KEY-STRING

The API key can be retrieved from the Key Manager startup logs or computed from the HSM.

minkms identity hsm:aes256:KEYVALUE

Unhealthy cluster

An unhealthy cluster is one where one ore more of the nodes are inaccessible from the rest of the cluster and no write requests can be processed.

If a node is temporarily or permanently lost, run minkms edit --rm to remove the node from the cluster configuration. After removing lost node(s), the remaining nodes resume write functionality.

Complete the following on a live node on the cluster that will remain part of the cluster.

Retrieve the ID for the unhealthy node.
```
minkms ls
```
Edit the cluster configuration to remove the unhealthy node.
```
minkms edit --rm NODE-ID --api-key ROOT-API-KEY-STRING
```
The API key can be retrieved from the Key Manager startup logs or computed from the HSM.
```
minkms identity hsm:aes256:KEYVALUE
```

Repeat the process for any other lost nodes.