TLS certificate troubleshooting

Diagnose and resolve common TLS certificate errors in MinIO AIStor deployments.

For general TLS setup, see the platform-specific network encryption guides under Installation. For cert-manager integration, see TLS certificate management with cert-manager.

x509: certificate signed by unknown authority

The side that opened the connection does not trust the certificate authority (CA) that signed the certificate it received. This applies both to an external client connecting to MinIO AIStor and to one AIStor node connecting to another.

Causes:

• The CA certificate is not in the client’s trust store
• The server certificate was signed by an intermediate CA, and the intermediate is missing from the chain
• On Kubernetes, the operator webhook certificate is signed by a different CA than what the MinIO pods trust
• A self-signed certificate was replaced on the nodes without restarting them, so the nodes no longer trust each other

Inter-node trust after a self-signed rotation:

This happens when the node certificates are their own trust anchors, which is what MinIO AIStor falls back to when CAs/ holds no authority that signed them. Such a certificate does two jobs at once: it is the certificate a node presents, and it is the trust anchor its peers use to verify that certificate. AIStor reloads the certificate it presents whenever the certificate files change, but it builds its CA trust pool once, at process start, and never rebuilds it. After you replace a self-signed certificate without restarting, each node therefore presents the new certificate while it still trusts only the old one, and new connections between nodes fail.

The trust check runs when a node opens a new connection to a peer, so connections that are already open keep working. The cluster loses inter-node connectivity as those connections drop and the nodes try to reconnect, which is why the impact grows after the rotation instead of appearing at the rotation itself.

Checking the certificate from outside the cluster does not reveal this. openssl s_client reports the certificate a node presents, which is the half that reloads on its own, so it shows the new certificate on every node while the nodes are already rejecting each other. A rotation that looks complete from a client can still be incomplete between nodes.

To stop this recurring, issue the node certificates from a certificate authority and place that authority in CAs/. The trust pool then holds the authority rather than the node certificates, so later rotations replace only public.crt and private.key and reload without a restart. See How MinIO AIStor reloads certificates.

Inter-node log signature:

The dialing node writes the following line to its error log, under the grid subsystem:

grid: marking https://HOST:PORT offline temporarily; caused by tls: failed to verify certificate: x509: certificate signed by unknown authority (STATE)

The peer appears as a full URL, and STATE is a numeric connection state, so a search anchored to either end of the line does not match. Search for marking and certificate signed by unknown authority together instead.

The line appears far less often than the failures do. AIStor counts the failures for each peer and logs the first three, then the 25th, the 50th, the 100th, and every 100th after that. The entries after the third carry a [last of N suppressed in D] suffix.

Because those thresholds spread out, a search of a long incident log returns only a handful of entries per peer, not one entry per minute. A small number of entries does not mean a small number of failures.

Resolution:

1. Identify the certificate’s issuer:

   openssl s_client -connect HOSTNAME:PORT -showcerts </dev/null 2>/dev/null | openssl x509 -noout -issuer -subject
   

2. Verify the CA is present in the MinIO trust store.

   For Linux deployments, place the CA certificate in the MinIO certs directory:

   cp ca.crt ~/.minio/certs/CAs/
   

   For Kubernetes deployments, verify the CA is mounted in the pod:

   kubectl exec -n NAMESPACE POD -c minio -- ls -la /tmp/minio/certs/CAs/
   

3. Verify the mounted CA matches the expected CA:

   kubectl exec -n NAMESPACE POD -c minio -- openssl x509 -in /tmp/minio/certs/CAs/ca-0.crt -noout -subject -issuer
   

   If the mounted certificate does not match the CA in the Kubernetes secret, the operator mount secret may be stale. See Stale mount secret below.

4. Restart AIStor after any change to the CA trust store, including adding a CA in the previous step. On Kubernetes, restart the pods. AIStor builds the CA trust pool once at startup and never rebuilds it, so a running node keeps using the pool it started with. If you replaced a self-signed certificate, restart every node, because each node’s certificate is also the CA its peers must trust. For the full procedure, see Certificate rotation.

x509: certificate has expired or is not yet valid

The server or CA certificate has passed its expiry date, or the system clock is incorrect.

Resolution:

1. Check the certificate expiry date:

   openssl s_client -connect HOSTNAME:PORT </dev/null 2>/dev/null | openssl x509 -noout -dates
   

2. Verify system clocks are synchronized across all nodes. MinIO AIStor requires clocks to be within 15 minutes of each other.

3. Renew the certificate. For cert-manager managed certificates, check the Certificate resource status:

   kubectl get certificate -n NAMESPACE -o wide
   

4. Monitor certificate expiry proactively using the Prometheus metric:

   minio_system_network_certificate_expires_in
   

x509: certificate is valid for X, not Y

The hostname used to connect does not match any Subject Alternative Name (SAN) in the certificate.

Resolution:

1. Inspect the certificate’s SANs:

   openssl s_client -connect HOSTNAME:PORT </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
   

2. Verify the hostname matches one of the listed SANs.

3. If using a load balancer or proxy, verify the certificate covers both the external hostname and the internal service names.

tls: bad certificate

or

remote error: tls: bad certificate

The TLS handshake failed because the client rejected the server’s certificate, or the server rejected the client’s certificate in an mTLS configuration.

Resolution:

1. Test connectivity with verbose TLS output:

   curl -v https://HOSTNAME:PORT 2>&1 | grep -A5 "SSL certificate"
   

2. For Kubernetes deployments, test from inside a MinIO pod to verify what the pod sees:

   kubectl exec -n NAMESPACE POD -c minio -- curl -v --cacert /tmp/minio/certs/CAs/ca-0.crt https://TARGET:PORT
   

3. If the test succeeds with --cacert but fails without it, the CA is not in the system or MinIO trust store.

Missing TLS password

or

Unable to decrypt the private key using the provided password

The private.key is password protected and AIStor cannot decrypt it at startup, either because no password was provided or because the supplied password is incorrect.

Resolution:

1. Set the MINIO_CERT_PASSWD environment variable to the private key’s password before starting the server:

   export MINIO_CERT_PASSWD=<PASSWORD>
   

2. Verify the encrypted key uses the PKCS-1 format. AIStor does not support OpenSSL’s default PKCS-8 format. Convert a PKCS-8 RSA key to PKCS-1:

   openssl rsa -in private-pkcs8-key.key -aes256 -passout pass:PASSWORD -out private.key
   

3. Confirm the certificate and key are in PEM format. PFX (.pfx) certificate bundles are not supported.

See Encrypted private keys for the full workflow.

The MinIO operator assembles all referenced certificates into a single mount secret named <objectstore-name>-generated in the tenant namespace. If the underlying certificate secrets are updated but the mount secret is not refreshed, the pods continue using outdated certificates.

Symptoms:

• The CA in the Kubernetes secret is correct, but the CA mounted in the pod is a different (older) certificate
• TLS errors persist after updating certificate secrets

Resolution:

1. Delete the mount secret to force the operator to rebuild it:

   kubectl delete secret OBJECTSTORE_NAME-generated -n NAMESPACE
   

2. The operator recreates the mount secret from the current certificate secrets and restarts the affected pods.

3. Verify the correct certificate is now mounted:

   kubectl exec -n NAMESPACE POD -c minio -- openssl x509 -in /tmp/minio/certs/CAs/ca-0.crt -noout -subject -issuer
   

The MinIO AIStor operator serves an upgrade webhook on port 4221. By default, the operator generates its own TLS certificate using the Kubernetes CSR API with the kubernetes.io/kubelet-serving signer. This certificate is signed by the cluster’s kubelet CA (CN=ca-kubelet), which MinIO pods may not trust.

Symptoms:

• MinIO version updates fail with x509: certificate signed by unknown authority when connecting to the operator webhook
• The operator log shows TLS handshake error from <IP>: remote error: tls: bad certificate

Resolution:

Disable operator autocert and provide a certificate signed by the same CA that the MinIO pods trust:

1. Create a cert-manager Certificate for the operator webhook:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: operator-webhook-tls
     namespace: OPERATOR_NAMESPACE
   spec:
     secretName: object-store-operator-tls
     issuerRef:
       name: YOUR_ISSUER
       kind: ClusterIssuer
     dnsNames:
       - object-store-operator
       - object-store-operator.OPERATOR_NAMESPACE.svc
       - object-store-operator.OPERATOR_NAMESPACE.svc.cluster.local
     duration: 8760h
     renewBefore: 720h
   

2. Set the following environment variables on the operator deployment:

   env:
     - name: OPERATOR_AUTO_CERT_DISABLED
       value: "on"
     - name: OPERATOR_CUSTOM_TLS_SECRET_NAME
       value: "object-store-operator-tls"
   

Use the following commands to diagnose TLS issues.

openssl s_client -connect HOSTNAME:PORT -showcerts </dev/null 2>/dev/null | openssl x509 -noout -text

openssl s_client -connect HOSTNAME:PORT </dev/null 2>/dev/null | openssl x509 -noout -enddate

openssl verify -CAfile ca.crt server.crt

kubectl exec -n NAMESPACE POD -c minio -- openssl s_client -connect TARGET:PORT -CAfile /tmp/minio/certs/CAs/ca-0.crt </dev/null

kubectl exec -n NAMESPACE POD -c minio -- find /tmp/minio/certs -name "*.crt" -exec openssl x509 -in {} -noout -subject -issuer -enddate \;

Query the minio_system_network_certificate_expires_in metric to find certificates approaching expiry:

minio_system_network_certificate_expires_in < 604800

This returns certificates expiring within 7 days (604800 seconds).

No command, admin API, or metric reports the CA trust pool that a running AIStor process holds. AIStor builds that pool when the process starts and does not expose it afterward.

The commands above read the certificate a node presents, or a certificate file on disk. Neither tells you which CAs the node accepts.

You can observe the certificates a running node currently holds, and you can observe inter-node connectivity:

• The minio_system_network_certificate_expires_in metric and the TLS section of the health report that mc support diag collects both read the in-memory certificate manager. They report the certificate that did reload. Neither reports the CA trust pool, so a clean result there proves nothing about peer trust.
• The minio_system_network_internode_peers_connected metric reports how many peers each node is connected to, not counting itself. Compare it against the number of peers you expect.

To confirm the on-disk state, compare the certificate files on every node, then restart AIStor on every node. Only a restart loads a changed CA trust pool.

• Enable TLS encryption for certificate file placement on Linux
• Certificate rotation for the certificate replacement procedure
• TLS certificate management with cert-manager for automated certificate provisioning
• System tuning checklist for NTP and time synchronization requirements