Operator webhook TLS

The MinIO AIStor Operator runs an internal upgrade webhook server on port 4221. MinIO pods connect to this webhook during in-pod version updates to download new binaries. This connection requires TLS, and the MinIO pods must trust the certificate presented by the operator.

How operator autocert works

By default, the operator generates its own TLS certificate using the Kubernetes Certificate Signing Request (CSR) API. The operator creates a CSR with the kubernetes.io/kubelet-serving signer, which the Kubernetes API server signs using the cluster’s kubelet CA.

The resulting certificate is stored in a secret named object-store-operator-tls in the operator namespace.

The operator also loads any secret prefixed with aistor-ca-tls in its namespace as a trusted CA. This allows the operator to trust certificates issued by external CAs.

When autocert causes trust failures

The kubelet CA that signs the operator’s certificate may not be trusted by the MinIO pods. This happens when:

  • The MinIO tenant uses disableAutoCert: true with certificates signed by a different CA
  • The operator certificate renews and gets signed by a CA that the MinIO pods do not trust
  • The cluster uses a non-standard CSR signer

Symptoms:

  • MinIO version updates fail with x509: certificate signed by unknown authority
  • The operator log shows TLS handshake error from <IP>: remote error: tls: bad certificate
  • The operator log shows Upgrade server Autocert is enabled, starting certificate setup

Using a custom certificate

Disable autocert and provide a certificate signed by the same CA that the MinIO pods trust.

Supported versions
The OPERATOR_CUSTOM_TLS_SECRET_NAME environment variable is available in Operator chart 5.2.0 and later (both aistor-objectstore-operator and aistor-operator charts). The global.operator.extraEnv Helm values path is consistent across all chart versions.

Step 1: Create a certificate

Create a cert-manager Certificate resource in the operator namespace:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: operator-webhook-tls
  namespace: OPERATOR_NAMESPACE
spec:
  secretName: object-store-operator-tls
  issuerRef:
    name: YOUR_ISSUER
    kind: ClusterIssuer
  dnsNames:
    - object-store-operator
    - object-store-operator.OPERATOR_NAMESPACE.svc
    - object-store-operator.OPERATOR_NAMESPACE.svc.cluster.local
  duration: 8760h
  renewBefore: 720h

Replace OPERATOR_NAMESPACE with the namespace where the operator is deployed (for example, objectstore-operator). Replace YOUR_ISSUER with the name of your cert-manager ClusterIssuer or Issuer.

Step 2: Disable autocert

Add the following environment variables to the operator deployment:

env:
  - name: OPERATOR_AUTO_CERT_DISABLED
    value: "on"
  - name: OPERATOR_CUSTOM_TLS_SECRET_NAME
    value: "object-store-operator-tls"

If using the Helm chart, set these under global.operator.extraEnv in the values file:

global:
  operator:
    extraEnv:
      - name: OPERATOR_AUTO_CERT_DISABLED
        value: "on"
      - name: OPERATOR_CUSTOM_TLS_SECRET_NAME
        value: "object-store-operator-tls"

Step 3: Restart the operator

The operator pod must restart to pick up the new environment variables and certificate:

kubectl rollout restart deployment object-store-operator -n OPERATOR_NAMESPACE

Step 4: Verify

Check the operator log for confirmation that the custom certificate is loaded:

kubectl logs -n OPERATOR_NAMESPACE deployment/object-store-operator | grep -i "custom certificate\|TLS"

Test the webhook endpoint from a MinIO pod:

kubectl exec -n TENANT_NAMESPACE POD -c minio -- \
    curl -v https://object-store-operator.OPERATOR_NAMESPACE.svc.cluster.local:4221/webhook/v1/update

A successful response returns HTTP 200 with a list of available MinIO binaries.

Admission webhook configuration

Starting with Operator chart 5.4.0, the admission webhook (used for pod scheduling and spread zone awareness) is configured as an object with additional options. This webhook is separate from the upgrade webhook on port 4221 described above.

The TLS environment variables (OPERATOR_AUTO_CERT_DISABLED, OPERATOR_CUSTOM_TLS_SECRET_NAME) and the global.operator.extraEnv Helm values path are the same across all chart versions, including the old aistor-objectstore-operator chart (4.x-5.1).

For general upgrade procedures, see Upgrade AIStor on Kubernetes.

Mount secret refresh

The operator assembles all referenced certificates (from certificates.server, certificates.client, and certificates.trustedCAs in the ObjectStore spec) into a single mount secret named <objectstore-name>-generated in the tenant namespace. This secret is mounted into the MinIO pods.

If the underlying certificate secrets are updated but the mount secret is not refreshed, the pods continue using outdated certificates.

To force a refresh:

kubectl delete secret OBJECTSTORE_NAME-generated -n TENANT_NAMESPACE

The operator recreates the mount secret from the current certificate secrets and restarts the affected pods.