Google Cloud Secret Manager

This tutorial shows how to setup a KES server that uses Google Cloud Secret Manager as a persistent key store.

Google Cloud Secret Manager

The Google Cloud Secret Manager is a key-value store for secrets, such as passwords, access tokens, and cryptographic keys.

1. Login in to the Google Cloud console

2. Create a new project or select an existing project

3. Enable the Secret Manager service if not already enabled for your project

4. Go to Google Cloud IAM for service accounts and create a new service account for KES. KES uses this service account to authenticate to GCP and access the Secret Manager.

5. Assign one or more roles to the new account

   If you want to get started quickly, assign the Secret Manager Admin role. However, this grants more permissions than KES needs.

   Alternatively, create a new role for KES with the minimal permissions required:

    secretmanager.secrets.create
    secretmanager.secrets.delete
    secretmanager.secrets.get

6. Create a key for the service account via Actions - Create Key.

   Use the JSON key format.

   Google Cloud allows you download a JSON file with the following structure:

   {
     "type":           "service_account",
     "project_id":     "<your-project-id>",
     "private_key_id": "<your-private-key-id>",
     "private_key":    "-----BEGIN PRIVATE KEY-----\n ... -----END PRIVATE KEY-----\n",
     "client_email":   "<your-service-account>@<your-project-id>.iam.gserviceaccount.com",
     "client_id":      "<your-client-id>"
   }

   Use this credentials file to configure KES to authenticate to Google Clout and access the Secret Manager.

KES Server setup

The KES Server requires a TLS private key and certificate.

The KES server is secure-by-default and can only run with TLS. This tutorial uses self-signed certificates for simplicity.

1. Generate a TLS private key and certificate for the KES server

   The following command generates a new TLS private key server.key and a self-signed X.509 certificate server.cert that is issued for the IP 127.0.0.1 and DNS name localhost (as SAN). Customize the command to match your setup.

   kes identity new --server --key server.key --cert server.cert --ip "127.0.0.1" --dns localhost

   Any other tooling for X.509 certificate generation works as well. For example, you could use openssl:

   $ openssl ecparam -genkey -name prime256v1 | openssl ec -out server.key
   
   $ openssl req -new -x509 -days 30 -key server.key -out server.cert \
       -subj "/C=/ST=/L=/O=/CN=localhost" -addext "subjectAltName = IP:127.0.0.1"

2. Create a private key and certificate for the application

   kes identity new --key=app.key --cert=app.cert app

   You can compute the app identity at any time.

   kes identity of app.cert

3. Create the config file server-config.yml

   address: 0.0.0.0:7373
   root:    disabled  # We disable the root identity since we don't need it in this guide 
   
   tls:
     key : server.key
     cert: server.cert
   
   policy:
     my-app:
       allow:
       - /v1/key/create/my-app*
       - /v1/key/generate/my-app*
       - /v1/key/decrypt/my-app*
       identities:
       - ${APP_IDENTITY}
   
   keystore:
     gcp:
       secretmanager:
         project_id: "<your-project-id>"                  # Use your GCP project ID
         credentials:
           client_email: "<your-client-email>"            # Use the client email from your GCP credentials file
           client_id: "<your-client-id>"                  # Use the client ID from your GCP credentials file
           private_key_id: "<your-private-key-id"         # Use the private key ID from your GCP credentials file
           private_key: "-----BEGIN PRIVATE KEY----- ..." # Use the private key from your GCP credentials file
   

4. Start a KES server in a new window/tab:

   Linux

   export APP_IDENTITY=$(kes identity of app.cert)
   
   kes server --config=server-config.yml --auth=off

   The command uses --auth=off because our root.cert and app.cert certificates are self-signed.

5. In the other window or tab, connect to the server

   export MINIO_KES_CERT_FILE=app.cert
   export MINIO_KES_KEY_FILE=app.key
   kes key create -k my-app-key

   Now, if you go to the Google Cloud Secret Manager, you should see a secret key named my-app-key.

6. Derive and decrypt data keys from the previously created my-app-key

   kes key dek -k my-app-key
   {
     plaintext : ...
     ciphertext: ...
   }

   kes key decrypt -k my-app-key <base64-ciphertext>

Using KES with a AIStor Server

AIStor Server requires KES to enable server-side data encryption.

See the KES for AIStor instruction guide for additional steps needed to use your new KES Server with a AIStor Server.

Configuration References

The following section describes the Key Encryption Service (KES) configuration settings to use Google Cloud Secret Manager as the root KMS to store external keys, such as the keys used for Server-Side Encryption on a AIStor Server.

address: 0.0.0.0:7373
root: ${ROOT_IDENTITY}

tls:
  key: kes-server.key
  cert: kes-server.cert

api:
  /v1/ready:
    skip_auth: false
    timeout:   15s

policy:
  minio-server:
    allow:
    - /v1/key/create/*
    - /v1/key/generate/*
    - /v1/key/decrypt/*
    - /v1/key/bulk/decrypt
    - /v1/key/list/*
    - /v1/status
    - /v1/metrics
    - /v1/log/audit
    - /v1/log/error
    deny:
    - /v1/key/generate/my-app-internal*
    - /v1/key/decrypt/my-app-internal*
    identities:
    - ${MINIO_IDENTITY}

  my-app:
    allow:
    - /v1/key/create/my-app*
    - /v1/key/generate/my-app*
    - /v1/key/decrypt/my-app*
    deny:
    - /v1/key/generate/my-app-internal*
    - /v1/key/decrypt/my-app-internal*
    identities:
    - df7281ca3fed4ef7d06297eb7cb9d590a4edc863b4425f4762bb2afaebfd3258
    - c0ecd5962eaf937422268b80a93dde4786dc9783fb2480ddea0f3e5fe471a731

keys:
  - name: "minio-encryption-key-alpha"
  - name: "minio-encryption-key-baker"
  - name: "minio-encryption-key-charlie"

cache:
  expiry:
    any: 5m0s
    unused: 20s
    offline: 0s

log:

  # Log error events to STDERR. Valid values are "on" or "off". 
  # Default is "on".
  error: on

  # Log audit events to STDOUT. Valid values are "on" and "off". 
  # Default is "off".
  audit: off

keystore:
  gcp:
    # https://cloud.google.com/secret-manager
    secretmanager:
      # See: https://cloud.google.com/resource-manager/docs/creating-managing-projects#before_you_begin
      project_id: ""
      endpoint: ""
      # Refer to: https://developers.google.com/identity/protocols/oauth2/scopes
      # If not set, the GCP default scopes are used.
      scopes: 
      - ""
      credentials:
        client_email: 
        client_id:    
        private_key_id: 
        private_key: