Certificate rotation

Rotate TLS certificates in MinIO AIStor deployments.

For certificate setup, see the platform-specific guides under Installation. For troubleshooting certificate errors, see TLS certificate troubleshooting.

MinIO AIStor uses two kinds of TLS certificate, and only one of them reloads while the server runs.

• Server certificates (public.crt and private.key) are the certificates a node presents to clients and to its peers. MinIO AIStor detects changes to these files and reloads them without a restart.
• CA certificates (the files in the CAs/ directory) are the certificate authority (CA) certificates a node uses to verify the servers it connects to, including other nodes in the same cluster. MinIO AIStor loads these only at startup and requires a restart to change them.

Rotate only the server certificate. Put a certificate authority in CAs/ once, issue every node’s public.crt from it, and leave CAs/ alone from then on. Rotation is then a replacement of public.crt and private.key, which MinIO AIStor reloads on its own, with no restart and no change to the trust pool. A certificate authority you generate yourself works for this; it does not have to come from a public CA.

Avoid making a server certificate its own trust anchor. If CAs/ does not contain an authority that signed the node certificates, each node falls back to trusting the server certificates it loaded at startup, and that fallback is what forces restarts:

• The trust pool is built once, at process start, and is never rebuilt.
• Replacing such a certificate updates only what the node presents, so the nodes stop trusting each other until every one of them restarts.
• Putting the node certificates themselves into CAs/ does not avoid this, because those files are then part of the trust pool and changing them needs a restart too.

If a deployment is in that state today, the way out is to issue node certificates from a certificate authority, place that authority in CAs/, and restart once. After that, rotations need no restart. See Rotating CA certificates for the rare case where the authority itself changes.

MinIO AIStor reloads server certificates as follows. Understanding the reload mechanism helps avoid common rotation mistakes.

• File replacement: MinIO AIStor watches certificate directories using inotify (Linux) for IN_CLOSE_WRITE and IN_MOVED_TO events. Write the new certificate and key to a temporary file, then rename (atomic move) into place.
• Symlink updates: For certificates managed by symlinks (common with cert-manager), MinIO AIStor polls symlink targets every 10 seconds.
• Content verification: MinIO AIStor computes SHA-256 hashes of the new content and only reloads if the content has changed.
• Failure safety: If the new certificate is invalid or partially written, MinIO AIStor continues serving the previously loaded certificate.

For details, see Automatic certificate reloading.

Verify the new certificate before deploying:

openssl x509 -in new-public.crt -noout -subject -issuer -dates -ext subjectAltName

Confirm:

• The subject and SANs match the MinIO server hostnames
• The issuer matches the expected CA
• The expiry date is in the future

Use atomic replacement to avoid serving a partial file:

cp new-public.crt /tmp/public.crt.tmp
cp new-private.key /tmp/private.key.tmp
mv /tmp/private.key.tmp ~/.minio/certs/private.key
mv /tmp/public.crt.tmp ~/.minio/certs/public.crt

Check that MinIO AIStor is serving the new certificate:

openssl s_client -connect HOSTNAME:9000 </dev/null 2>/dev/null | openssl x509 -noout -enddate -serial

Compare the serial number and expiry with the new certificate file to confirm the reload succeeded.

mc admin info ALIAS

If the CA itself changed (not just the server certificate), place the new CA certificate in the CAs/ directory on every node:

cp new-ca.crt ~/.minio/certs/CAs/

A deployment whose node certificates are their own trust anchors needs the same restart, because each node trusts the server certificate it loaded at startup. Replace public.crt and private.key on every node as described in Step 2, then restart every node.

Rather than repeat that on every rotation, move the deployment onto a certificate authority: generate one, issue each node’s certificate from it, place the authority in CAs/ on every node, and restart once. Later rotations then replace only public.crt and private.key, and MinIO AIStor reloads them without a restart.

Using the AIStor Client (restarts all nodes):

mc admin service restart ALIAS

Or restart the service on individual nodes:

sudo systemctl restart minio

cert-manager handles rotation automatically based on the renewBefore setting in the Certificate resource. No manual intervention is required.

Verify the Certificate resource is renewing correctly:

kubectl get certificate -n NAMESPACE -o wide

The READY column should show True and RENEWAL should indicate the next renewal time.

If a certificate is stuck in a not-ready state:

kubectl describe certificate CERT_NAME -n NAMESPACE

Check the Events section for errors from cert-manager, such as an unreachable issuer or an authorization failure.

For certificates provided via Kubernetes secrets (certificates.server in the ObjectStore spec):

1. Update the secret with the new certificate and key:

   kubectl create secret tls CERT_SECRET_NAME \
       --cert=new-public.crt --key=new-private.key \
       -n NAMESPACE --dry-run=client -o yaml | kubectl apply -f -
   

2. Delete the operator mount secret to force a refresh:

   kubectl delete secret OBJECTSTORE_NAME-generated -n NAMESPACE
   

   The operator recreates the mount secret from the current certificate secrets and restarts the pods.

3. Verify the new certificate is mounted:

   kubectl exec -n NAMESPACE POD -c minio -- openssl x509 -in /tmp/minio/certs/public.crt -noout -enddate -serial
   

The operator webhook certificate is separate from the tenant certificates. See Operator webhook TLS for rotation procedures.

MinIO AIStor exposes the minio_system_network_certificate_expires_in gauge metric (metrics v3 only), which reports the number of seconds until each certificate expires.

Alert on certificates expiring within 30 days:

minio_system_network_certificate_expires_in < 2592000

Alert on certificates expiring within 7 days (critical):

minio_system_network_certificate_expires_in < 604800

Check expiry from the command line:

openssl s_client -connect HOSTNAME:9000 </dev/null 2>/dev/null | openssl x509 -noout -enddate

For Kubernetes, check all certificates mounted in a pod:

kubectl exec -n NAMESPACE POD -c minio -- \
    find /tmp/minio/certs -name "*.crt" -exec openssl x509 -in {} -noout -subject -enddate \;

• Automatic certificate reloading for reload mechanism details
• TLS certificate management with cert-manager for automated provisioning
• TLS certificate troubleshooting for diagnosing errors after rotation
• Operator webhook TLS for operator certificate rotation