TLS certificate management with cert-manager

The MinIO AIStor Operator supports cert-manager to provision and manage certificates, as an alternative to the Operator. cert-manager obtains valid certificates from an Issuer or ClusterIssuer and can automatically renew certificates prior to expiration.

A ClusterIssuer issues certificates for multiple Namespaces. An Issuer only mints certificates for its own Namespace.

You complete the following steps to manage your TLS certificates with cert-manager:

1. Install cert-manager

2. Create cluster issuers (self-signed and with fixed CA root certificate)

3. Create cert-manager certificates for each object store.

4. Add certificates with cert-manager to object stores.

5. Optionally, issue the Operator’s own certificate with cert-manager (required on clusters that do not sign kubernetes.io/kubelet-serving CSRs).

1. Install cert-manager. See the cert-manager documentation for details. Version 1.20 is recommended.

2. Create a Cluster Issuer resource for your cluster that can generate self-signed certificates. This is needed to create the initial self-signed certificate that is used by the CA.

   # selfsigned-issuer.yaml
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: selfsigned-issuer
   spec:
     selfSigned: {}
   

3. Create the self-signed root CA certificate using the selfsigned-issuer issuer, by creating the root-ca certificate in the cert-manager namespace that is valid for 10 years:

   # root-ca.yaml
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: root-ca
     namespace: cert-manager      # change if cert-manager is installed in another namespace
   spec:
     commonName: root-ca
     duration: 87600h        # 10 years
     renewBefore: 720h       # 30 days
     isCA: true
     issuerRef:
       group: cert-manager.io
       kind: ClusterIssuer
       name: selfsigned-issuer
     privateKey:
       algorithm: ECDSA
       size: 256
     secretName: root-ca
   

4. Create a Cluster Issuer resource for your cluster that issues certificates that are signed using the root-ca certificate:

   # ca-issuer.yaml
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: ca-issuer
   spec:
     ca:
       secretName: root-ca
   

   This cluster issuer will always sign certificates using the root-ca certificate.

5. Create the certificate for the object store with cert-manager.

   The certificate must be valid for the following DNS domains:

   • minio.<namespace>, minio.<namespace>.svc and minio.<namespace>.svc.<cluster domain> (S3 API)
   • *.<object store-name>-hl.<namespace>.svc.<cluster domain> (inter-node service)
   • Optional <object store-name>-console.<namespace>, <object store-name>-console.<namespace>.svc and <object store-name>-console.<namespace>.svc.<cluster domain> (console)

   where:

   • <cluster domain> is the internal root DNS domain assigned in your Kubernetes cluster. Typically, this is cluster.local, but confirm the value by checking your CoreDNS configuration for the correct value for your Kubernetes cluster.

     Different Kubernetes providers manage the root domain differently. Check with your Kubernetes provider for more information.

   • <object store-name> is the name provided to your object store in the metadata.name of the object store YAML. For this example it is myaistor.

   • <namespace> is the value created earlier where the object store will be installed. In the object store YAML, it is defined in the the metadata.namespace field. For this example the value is object-store-example.

   Create a file called object-store-example-minio-certificate.yaml (or whatever suits your naming conventions). The contents of the file should resemble the following, modified to reflect your cluster and object store configurations:

   # object-store-example-minio-certificate.yaml
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: object-store-certmanager-cert
     namespace: object-store-example
   spec:
     dnsNames:
       # S3 API (required)
       - "minio.object-store-example"
       - "minio.object-store-example.svc"
       - 'minio.object-store-example.svc.cluster.local'
       # Internode service (headless, required)
       - '*.myaistor-hl.object-store-example.svc.cluster.local'
       # AIStor console (optional)
       - "myaistor-console.object-store-example"
       - "myaistor-console.object-store-example.svc"
       - 'myaistor-console.object-store-example.svc.cluster.local'
     secretName: myaistor-tls
     issuerRef:
       group: cert-manager.io
       kind: ClusterIssuer
       name: ca-issuer
   

   Tip

   For this example, the object store name is myaistor. We recommend naming the secret in the field spec.secretName as <object store-name>-tls as a naming convention.

When deploying an object store, you must set the TLS configuration such that:

• The object store does not automatically generate its own certificates (spec.certificates.disableAutoCert: true) and

• The object store has a valid cert-manager reference (spec.certificates.server)

This directs the Operator to deploy the object store using the cert-manager certificates exclusively.

The following YAML spec provides a baseline configuration meeting these requirements:

apiVersion: aistor.min.io/v1
kind: ObjectStore
metadata:
  name: myaistor
  namespace: object-store-example
spec:
...
  certificates:
    disableAutoCert: true    # leave this to `false` if you only use cert-manager to create external certificates
    server:
    - name: myaistor-tls
      type: kubernetes.io/tls
...

The steps above configure cert-manager for the object store. The AIStor Operator also presents its own TLS certificate for its internal upgrade service, which MinIO pods connect to during in-pod version updates. By default the Operator generates this certificate through the Kubernetes certificate signing request (CSR) API using the kubernetes.io/kubelet-serving signer.

On a cluster where no component signs kubernetes.io/kubelet-serving requests, for example a cluster that relies on cert-manager with the in-tree signer disabled, this certificate is never issued. The CSR stays in the Approved condition but no certificate is produced, and the Operator cannot start its upgrade service. In this case, provide the Operator certificate from cert-manager instead.

Reuse the ca-issuer created earlier to issue a certificate for the Operator service. Create the certificate in the namespace where the Operator runs and store it in a secret named object-store-operator-tls:

# operator-certmanager-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: operator-certmanager-cert
  namespace: aistor
spec:
  secretName: object-store-operator-tls
  dnsNames:
    - object-store-operator
    - object-store-operator.aistor.svc
    - object-store-operator.aistor.svc.cluster.local
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: ca-issuer

Replace aistor with the namespace where the Operator runs. The certificate needs only the DNS names of the Operator service. It does not require the system:node common name that the autocert CSR uses, because the Operator reads the certificate directly from the secret instead of through the CSR signer.

Set the following environment variables on the Operator. With the Helm chart, add them under global.operator.extraEnv in the Operator values file (aistor-operator-values.yaml, the same file you used to install the Operator):

global:
  operator:
    extraEnv:
      - name: OPERATOR_AUTO_CERT_DISABLED
        value: "on"
      - name: OPERATOR_CUSTOM_TLS_SECRET_NAME
        value: "object-store-operator-tls"

OPERATOR_AUTO_CERT_DISABLED turns off the Operator’s automatic CSR-based certificate. OPERATOR_CUSTOM_TLS_SECRET_NAME tells the Operator to read its certificate from the secret that cert-manager populates.

Apply the updated values file with helm upgrade. Updating the environment variables rolls the Operator deployment automatically, so a manual restart is not required:

helm upgrade aistor minio/aistor-operator -n aistor -f aistor-operator-values.yaml

Replace the release name (aistor), chart (minio/aistor-operator), namespace (aistor), and values file (aistor-operator-values.yaml) with the values you used to install the Operator.

You do not need to restart the Operator again when the certificate is later renewed. The Operator watches the secret and reloads the renewed certificate automatically, within about a minute of cert-manager updating it.

During an in-pod upgrade, MinIO connects to the Operator’s upgrade service and verifies the certificate it presents. Because that certificate is issued by cert-manager, the object store must trust the issuing CA, otherwise the TLS verification fails and the upgrade cannot complete.

Copy the issuing CA from the Operator certificate secret (object-store-operator-tls) into a secret in the object store namespace, with the certificate stored under the key public.crt:

kubectl get secret object-store-operator-tls -n aistor \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > ca-issuer-ca.crt

kubectl create secret generic ca-issuer-ca \
  --from-file=public.crt=ca-issuer-ca.crt \
  -n object-store-example

Then add the secret to the object store’s spec.certificates.trustedCAs and apply the object store:

spec:
  certificates:
    trustedCAs:
      - name: ca-issuer-ca
        type: opaque

Replace aistor, object-store-example, and the secret name with the values for your deployment. The Operator mounts the CA into the MinIO pods, so the certificate presented by the upgrade service is trusted.

For the complete procedure, including how to verify the certificate from a MinIO pod, see Operator webhook TLS.