Certificate Management

AIStor Key Manager uses TLS certificates to secure all network connections, including client connections and inter-node communication within a cluster. Proper certificate management maintains secure operations and prevents service disruptions.

Quick reference

Task Action Notes
Check certificate expiration Run openssl command on certificate file Shows validity period
Update certificate files Replace files at configured paths Reload happens automatically within 60 minutes
Force immediate reload Restart the server For urgent updates or maintenance windows
View reload logs Check server logs Monitor for expiration warnings
Automatic reload interval Every 60 minutes No manual action required
Expiration warning threshold 48 hours before expiry Monitor logs for warnings

To check certificate expiration, run:

openssl x509 -in /etc/minkms/public.crt -noout -dates
Version Requirement
Automatic certificate reloading requires Key Manager version RELEASE.2025-07-15T06-12-56Z or later.

Automatic TLS certificate refresh

Starting with RELEASE.2025-07-15T06-12-56Z, Key Manager automatically reloads TLS certificates every 60 minutes without requiring a server restart. Previous versions store the certificate at server start, requiring a manual restart of the Key Manager cluster to load new certificates. This feature simplifies certificate management for renewals and rotations.

Reload behavior

The automatic reload process:

  1. Checks all certificate and private key files configured in the tls.certs section of the configuration file.
  2. Reloads each certificate-key pair from the local storage.
  3. Logs warnings for any certificates that fail to load or that expire within 48 hours.
  4. Continues using existing certificates if reload fails.

The reload interval of 60 minutes is sufficient for most operational scenarios, as certificates typically have validity periods measured in days, months, or years.

Monitoring certificate status

Key Manager logs certificate-related events during startup and automatic reload cycles.

The server performs these monitoring operations:

  • Periodic reload: Reloads all configured TLS certificates every 60 minutes.
  • Expiration warnings: Logs warnings for certificates that expire within 48 hours.
  • Reload failure alerts: Logs warnings when a certificate or private key fails to reload.

Certificate expiration warnings appear in the server logs with messages similar to:

Certificate /etc/minkms/public.crt expires in 36h
Certificate /etc/minkms/public.crt expired on 2025-01-15 14:30:00
Failed to read server certificate: open /etc/minkms/public.crt: no such file or directory

Retrieve logs by running journalctl -u minkms on any Key Manager node or using the minkms logs command.

Update certificates

Certificate requirements

TLS certificates used with Key Manager must meet the following requirements:

  • X.509 format certificates.
  • Supported public key algorithms: RSA-2048, RSA-3072, RSA-4096, ECDSA P-256, ECDSA P-384, Ed25519.
  • Certificate must be valid (not expired).
  • Certificate must include the server hostname or IP address in the Subject Alternative Name (SAN) field.
  • Private key must be unencrypted (not password-protected).
Performance Recommendation
RSA certificates can negatively impact performance. Key Manager logs a warning if it detects that any configured certificate uses an RSA algorithm. Use ECDSA or EdDSA certificates for better performance.

For detailed TLS requirements, see TLS Certificates.

Update procedure

Update certificates in a rolling fashion across cluster nodes. The automatic reload operates on each node independently, allowing you to update nodes one at a time. Verify successful certificate refresh on each node before proceeding to the next.

To update TLS certificates:

  1. Obtain new certificates from your certificate authority.

  2. Verify new certificates match the requirements above and include correct hostnames in their SANs.

  3. Replace certificate files at the configured paths. Ensure the Key Manager process has read permissions for the certificate files.

  4. Choose reload method:

    • Automatic reload: Wait up to 60 minutes for the next reload cycle.
    • Immediate reload: Restart the Key Manager server.

  5. Verify successful reload by checking the server logs for reload messages.

Certificate Synchronization in Clusters
In multi-node clusters, ensure all nodes have valid certificates with matching hostnames in their SANs. Certificate expiration on any node can disrupt cluster consensus and availability.

Troubleshooting

Certificate refresh failures

Symptom: Log messages like Failed to read server certificate: open /etc/minkms/public.crt: no such file or directory

Cause: Certificate files do not exist at configured paths, have incorrect permissions, or are invalid.

Resolution:

  1. Verify the certificate and private key files exist at the paths specified in the tls.certs configuration section.
  2. Confirm the Key Manager process has read permissions for the certificate files.
  3. Verify the certificate and private key are valid X.509 format.
  4. Wait for the next reload cycle or restart the server.

The server continues operating with existing certificates if reload fails.

Certificate expiration

Symptom: Log message Certificate /etc/minkms/public.crt expired on 2025-01-15 14:30:00

Cause: Certificate has passed its expiration date.

Impact:

  • New client connections fail with TLS handshake errors.
  • In cluster deployments, inter-node communication may fail.

Resolution:

  1. Obtain renewed certificates immediately.
  2. Replace the expired certificate files.
  3. Restart the server for immediate reload.
  4. Verify successful reload in the server logs.

Cluster communication issues after certificate update

Symptom: Cluster nodes cannot communicate, write operations fail, consensus lost.

Cause: Certificates not updated on all nodes, or certificates have incorrect hostnames in SANs.

Resolution:

  1. Verify all cluster nodes have valid, non-expired certificates.
  2. Verify each node’s certificate includes the correct hostname in the SAN field.
  3. Update certificates on any nodes with invalid or expired certificates.
  4. Restart nodes that have certificate issues.
  5. Monitor cluster logs for successful node rejoining.

Platform-specific considerations

For detailed platform-specific installation and configuration, see the Installation section.