Cluster Deployment

This guide walks through deploying a multi-node MinIO KMS cluster on Linux from scratch. A clustered deployment provides high availability for encryption key management — any node can serve read requests, and the cluster replicates all state (enclaves, identities, keys) automatically.

For production environments, deploy a minimum of 3 nodes. See Fault Tolerance for details on read and write availability.

All nodes in a cluster must use the same HSM key. The HSM key establishes root trust between nodes and encrypts the on-disk state. Losing the HSM key results in permanent data loss for the entire cluster. Store the key in a secure location such as a dedicated secrets vault.

Prerequisites

3 or more Linux hosts (AMD64 or ARM64) with network connectivity between them
TLS certificates for each host (private key + public certificate signed by a trusted CA)
DNS resolution or /etc/hosts entries so each node can resolve the others by hostname

If you have not yet installed MinIO KMS on any host, complete steps 1 through 6 of the Linux installation guide on each node before proceeding. Use the same HSM key on all nodes.

Procedure

1. Generate a shared HSM key

Generate the HSM key on any one host. Use the same key on all nodes in the cluster.

minkms --soft-hsm

The command outputs a key in the format hsm:aes256:KEYVALUE. Store this value securely — you need it for every node.

2. Configure each node

On each node, set the environment file at /etc/default/minkms:

MINIO_KMS_HSM_KEY=hsm:aes256:KEYVALUE

MINIO_KMS_VOLUME=/mnt/minio-kms

MINIO_KMS_OPTS="--host HOSTNAME:7373 --config /etc/minkms/config.yaml"

Replace KEYVALUE with the HSM key generated in step 1.

Replace HOSTNAME with the resolvable hostname of the current node (for example, minkms-node0.example.net).

The --host flag is critical for multi-node deployments. It tells MinIO KMS to identify itself using the specified hostname rather than the local IP address. The hostname you provide must match the hostname you use later in the minkms add command. A mismatch causes a “server not part of cluster” error on startup.

Create the configuration file at /etc/minkms/config.yaml:

version: v1

tls:
  certs:
    - key: /etc/minkms/certs/private.key
      cert: /etc/minkms/certs/public.crt
  ca: /etc/minkms/certs/CAs

3. Start all nodes

On each node, enable and start the MinIO KMS service:

systemctl daemon-reload
systemctl enable minkms
systemctl start minkms

Verify each node starts successfully:

journalctl -u minkms -n 20

Each node starts as an independent single-node cluster. The output shows the node’s endpoint and API key:

Version        2025-11-12T19-14-51Z
HSM            hsm:minio:soft
Cluster        ID      a8780bd2-a31c-49dd-953b-2c7619ae3249
               Node 0: minkms-node0.example.net:7373 <-

Endpoint       https://minkms-node0.example.net:7373
API Key        k1:APIKEY

=> Server is up and running...

All nodes that share the same HSM key produce the same API key. Store this key — you need it for the remaining steps.

4. Form the cluster

Choose one node as the initial cluster node (for example, minkms-node0). From that node, add each additional node:

export MINIO_KMS_SERVER=https://minkms-node0.example.net:7373
export MINIO_KMS_API_KEY=k1:APIKEY

minkms add minkms-node1.example.net:7373
minkms add minkms-node2.example.net:7373

MINIO_KMS_SERVER and MINIO_KMS_API_KEY are CLI environment variables for running minkms commands. They are not part of the server configuration in /etc/default/minkms. Set them in your shell session or prefix them to the command.

The minkms add command connects to the existing cluster at MINIO_KMS_SERVER and tells it to add the specified node. The new node must be:

Running and reachable from the existing cluster
A fresh single-node cluster (not already part of another cluster)
Using the same HSM key

If the node was previously part of a different cluster, stop MinIO KMS, delete the database, and restart:

systemctl stop minkms
rm /mnt/minio-kms/kms.db
systemctl start minkms

5. Verify the cluster

Check the cluster status from any node:

export MINIO_KMS_SERVER=https://minkms-node0.example.net:7373
export MINIO_KMS_API_KEY=k1:APIKEY

minkms stat

A healthy 3-node cluster displays:

●  Node 0 at minkms-node0.example.net:7373
   Version     2025-11-12T19-14-51Z
   Uptime      5m
   State       Leader
   Leader      Node 0

●  Node 1 at minkms-node1.example.net:7373
   Version     2025-11-12T19-14-51Z
   Uptime      4m
   State       Follower
   Leader      Node 0

●  Node 2 at minkms-node2.example.net:7373
   Version     2025-11-12T19-14-51Z
   Uptime      3m
   State       Follower
   Leader      Node 0

All [3/3] nodes are up ✔

Verify that:

All nodes show as up
One node is Leader, the rest are Follower
Commit values are identical across all nodes

6. Create an enclave and identity

Once the cluster is formed, create an enclave and identity for AIStor. Run this once from any node — the cluster replicates the state to all nodes automatically.

export MINIO_KMS_SERVER=https://minkms-node0.example.net:7373
export MINIO_KMS_API_KEY=k1:APIKEY

minkms add-enclave aistor-primary
minkms add-identity --enclave aistor-primary --admin

The add-identity command outputs an API key for the AIStor object store. Store this key securely.

Verify the key replicated to all nodes by querying from a different node:

export MINIO_KMS_SERVER=https://minkms-node2.example.net:7373

minkms ls-key --enclave aistor-primary

7. Configure AIStor

Update the AIStor environment to point to the MinIO KMS cluster. Specify all KMS nodes as a comma-separated list:

Environment Variable	Value
`MINIO_KMS_SERVER`	`https://minkms-node0.example.net:7373,https://minkms-node1.example.net:7373,https://minkms-node2.example.net:7373`
`MINIO_KMS_API_KEY`	The `k1:` prefixed enclave admin API key from step 6
`MINIO_KMS_ENCLAVE`	`aistor-primary`
`MINIO_KMS_SSE_KEY`	Name of the default encryption key (create one with `minkms add-key --enclave aistor-primary sse-key`)

Restart the AIStor service on each node to apply the changes:

sudo systemctl restart minio

Do not use mc admin service restart for this step. That command restarts AIStor with the environment it started with, so AIStor does not load the new settings.

Troubleshooting

“server not part of cluster” on startup

The --host value in /etc/default/minkms does not match the hostname used in minkms add. For example, if you ran minkms add minkms-node1.example.net:7373 but the node starts with --host 10.0.0.2:7373, MinIO KMS cannot match itself to the cluster configuration.

Fix: Update MINIO_KMS_OPTS in /etc/default/minkms to use the same hostname, then stop the service, delete the database, and restart:

systemctl stop minkms
rm /mnt/minio-kms/kms.db
systemctl start minkms

Then re-add the node from an existing cluster member.

“failed to unseal root encryption key with HSM: invalid ciphertext”

The node has an existing database (kms.db) sealed with a different HSM key. This occurs when the node was previously started with a different key.

Fix: Stop MinIO KMS, delete the old database, verify the HSM key in /etc/default/minkms, and restart:

systemctl stop minkms
rm /mnt/minio-kms/kms.db
systemctl start minkms

“access denied: insufficient permissions”

The CLI environment variables are incorrect or the MINIO_KMS_SERVER points to the wrong node. Verify:

echo $MINIO_KMS_SERVER
echo $MINIO_KMS_API_KEY

The server URL must include https:// and the API key must match the cluster’s root key.

Node shows as down in `minkms stat`

Verify network connectivity between nodes:

curl -k https://minkms-node1.example.net:7373/version

If the node is reachable but still shows as down, check if it has the correct --host value and restart it.

Next Steps

Scale the cluster by adding or removing nodes
Back up and recover the cluster state
Configure a load balancer in front of the cluster for client connections

Cluster Deployment

On this page