TLS certificate troubleshooting

This guide covers common TLS certificate errors in MinIO AIStor deployments and provides procedures for diagnosing and resolving them.

For general TLS setup, see the platform-specific network encryption guides under Installation. For cert-manager integration, see TLS certificate management with cert-manager.

x509: certificate signed by unknown authority

The server presented a certificate signed by a CA that the client does not trust.

Causes:

• The CA certificate is not in the client’s trust store
• The server certificate was signed by an intermediate CA, and the intermediate is missing from the chain
• On Kubernetes, the operator webhook certificate is signed by a different CA than what the MinIO pods trust

Resolution:

1. Identify the certificate’s issuer:

   openssl s_client -connect HOSTNAME:PORT -showcerts </dev/null 2>/dev/null | openssl x509 -noout -issuer -subject
   

2. Verify the CA is present in the MinIO trust store.

   For Linux deployments, place the CA certificate in the MinIO certs directory:

   cp ca.crt ~/.minio/certs/CAs/
   

   For Kubernetes deployments, verify the CA is mounted in the pod:

   kubectl exec -n NAMESPACE POD -c minio -- ls -la /tmp/minio/certs/CAs/
   

3. Verify the mounted CA matches the expected CA:

   kubectl exec -n NAMESPACE POD -c minio -- openssl x509 -in /tmp/minio/certs/CAs/ca-0.crt -noout -subject -issuer
   

   If the mounted certificate does not match the CA in the Kubernetes secret, the operator mount secret may be stale. See Stale mount secret below.

x509: certificate has expired or is not yet valid

The server or CA certificate has passed its expiry date, or the system clock is incorrect.

Resolution:

1. Check the certificate expiry date:

   openssl s_client -connect HOSTNAME:PORT </dev/null 2>/dev/null | openssl x509 -noout -dates
   

2. Verify system clocks are synchronized across all nodes. MinIO AIStor requires clocks to be within 15 minutes of each other.

3. Renew the certificate. For cert-manager managed certificates, check the Certificate resource status:

   kubectl get certificate -n NAMESPACE -o wide
   

4. Monitor certificate expiry proactively using the Prometheus metric:

   minio_system_network_certificate_expires_in
   

x509: certificate is valid for X, not Y

The hostname used to connect does not match any Subject Alternative Name (SAN) in the certificate.

Resolution:

1. Inspect the certificate’s SANs:

   openssl s_client -connect HOSTNAME:PORT </dev/null 2>/dev/null | openssl x509 -noout -ext subjectAltName
   

2. Verify the hostname matches one of the listed SANs.

3. If using a load balancer or proxy, verify the certificate covers both the external hostname and the internal service names.

tls: bad certificate

or

remote error: tls: bad certificate

The TLS handshake failed because the client rejected the server’s certificate, or the server rejected the client’s certificate in an mTLS configuration.

Resolution:

1. Test connectivity with verbose TLS output:

   curl -v https://HOSTNAME:PORT 2>&1 | grep -A5 "SSL certificate"
   

2. For Kubernetes deployments, test from inside a MinIO pod to verify what the pod sees:

   kubectl exec -n NAMESPACE POD -c minio -- curl -v --cacert /tmp/minio/certs/CAs/ca-0.crt https://TARGET:PORT
   

3. If the test succeeds with --cacert but fails without it, the CA is not in the system or MinIO trust store.

The MinIO operator assembles all referenced certificates into a single mount secret named <objectstore-name>-generated in the tenant namespace. If the underlying certificate secrets are updated but the mount secret is not refreshed, the pods continue using outdated certificates.

Symptoms:

• The CA in the Kubernetes secret is correct, but the CA mounted in the pod is a different (older) certificate
• TLS errors persist after updating certificate secrets

Resolution:

1. Delete the mount secret to force the operator to rebuild it:

   kubectl delete secret OBJECTSTORE_NAME-generated -n NAMESPACE
   

2. The operator recreates the mount secret from the current certificate secrets and restarts the affected pods.

3. Verify the correct certificate is now mounted:

   kubectl exec -n NAMESPACE POD -c minio -- openssl x509 -in /tmp/minio/certs/CAs/ca-0.crt -noout -subject -issuer
   

The MinIO AIStor operator serves an upgrade webhook on port 4221. By default, the operator generates its own TLS certificate using the Kubernetes CSR API with the kubernetes.io/kubelet-serving signer. This certificate is signed by the cluster’s kubelet CA (CN=ca-kubelet), which MinIO pods may not trust.

Symptoms:

• MinIO version updates fail with x509: certificate signed by unknown authority when connecting to the operator webhook
• The operator log shows TLS handshake error from <IP>: remote error: tls: bad certificate

Resolution:

Disable operator autocert and provide a certificate signed by the same CA that the MinIO pods trust:

1. Create a cert-manager Certificate for the operator webhook:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: operator-webhook-tls
     namespace: OPERATOR_NAMESPACE
   spec:
     secretName: object-store-operator-tls
     issuerRef:
       name: YOUR_ISSUER
       kind: ClusterIssuer
     dnsNames:
       - object-store-operator
       - object-store-operator.OPERATOR_NAMESPACE.svc
       - object-store-operator.OPERATOR_NAMESPACE.svc.cluster.local
     duration: 8760h
     renewBefore: 720h
   

2. Set the following environment variables on the operator deployment:

   env:
     - name: OPERATOR_AUTO_CERT_DISABLED
       value: "on"
     - name: OPERATOR_CUSTOM_TLS_SECRET_NAME
       value: "object-store-operator-tls"
   

Use the following commands to diagnose TLS issues.

openssl s_client -connect HOSTNAME:PORT -showcerts </dev/null 2>/dev/null | openssl x509 -noout -text

openssl s_client -connect HOSTNAME:PORT </dev/null 2>/dev/null | openssl x509 -noout -enddate

openssl verify -CAfile ca.crt server.crt

kubectl exec -n NAMESPACE POD -c minio -- openssl s_client -connect TARGET:PORT -CAfile /tmp/minio/certs/CAs/ca-0.crt </dev/null

kubectl exec -n NAMESPACE POD -c minio -- find /tmp/minio/certs -name "*.crt" -exec openssl x509 -in {} -noout -subject -issuer -enddate \;

Query the minio_system_network_certificate_expires_in metric to find certificates approaching expiry:

minio_system_network_certificate_expires_in < 604800

This returns certificates expiring within 7 days (604800 seconds).

• Enable TLS encryption for certificate file placement on Linux
• TLS certificate management with cert-manager for automated certificate provisioning
• System tuning checklist for NTP and time synchronization requirements