Server-Side Object Encryption with AWS Root KMS

MinIO Server-Side Encryption (SSE) protects objects as part of write operations, allowing clients to take advantage of server processing power to secure objects at the storage layer (encryption-at-rest). SSE also provides key functionality to regulatory and compliance requirements around secure locking and erasure.

MinIO SSE uses Key Encryption Service (KES) and an external root Key Management Service (KMS) for performing secured cryptographic operations at scale. The root KMS provides stateful and secured storage of External Keys (EK) while KES is stateless and derives additional cryptographic keys from the root-managed EK.

This procedure does the following:

  • Configure KES to use AWS Secrets Manager as the root KMS.

  • Configure MinIO to use the KES instance for supporting SSE.

  • Configure automatic bucket-default SSE-KMS and SSE-S3.

Prerequisites

AWS Key Management Service

This procedure assumes familiarity with AWS Key Management Service and AWS Secrets Manager. The Getting Started with AWS Key Management Service provides a sufficient foundation for the purposes of this procedure.

MinIO specifically requires the following AWS settings or configurations:

  • A new AWS Programmatic Access user with corresponding access key and secret key.

  • A policy that grants the created user access to AWS Secrets Manager and AWS KMS. The following policy grants the minimum necessary permissions:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "minioSecretsManagerAccess",
          "Action": [
            "secretsmanager:CreateSecret",
            "secretsmanager:DeleteSecret",
            "secretsmanager:GetSecretValue",
            "secretsmanager:ListSecrets"
          ],
          "Effect": "Allow",
          "Resource": "*"
        },
        {
          "Sid": "minioKmsAccess",
          "Action": [
            "kms:Decrypt",
            "kms:DescribeKey",
            "kms:Encrypt"
          ],
          "Effect": "Allow",
          "Resource": "*"
        }
      ]
    }
    

    AWS provides the SecretsManagerReadWrite and AWSKeyManagementServicePowerUser canned roles that meet and exceed the minimum required permissions.

Network Encryption (TLS)

MinIO Key Encryption Service (KES) relies on mutual TLS (mTLS) for authentication and authorization. Enabling SSE therefore requires that the MinIO server, KES, and the root KMS enforce TLS.

The instructions on this page include creation of TLS certificates for supporting mTLS between MinIO and the KES instance. These certificates are appropriate for early development and evaluation environments only.

For instructions on enabling TLS on the MinIO server, see Network Encryption (TLS).

Use Caution in Production Environments

DO NOT use the TLS certificates generated as part of this procedure for any long-term development or production environments.

Defer to organization/industry best practices around TLS certificate generation and management. A complete guide to creating valid certificates (e.g. well-formed, current, and trusted) is beyond the scope of this procedure.

Podman Container Manager

The procedures on this page use the Podman daemonless container engine for running OCI containers. Podman is a drop-in replacement for Docker.

This procedure assues running Podman in “rootless”. Defer to the Podman documentation for installing and configuring Podman to run in rootless mode.

Enable MinIO Server-Side Encryption with AWS Root KMS

The following steps deploy Key Encryption Service (KES) configured to use an existing AWS KMS and Secrets Manager deployment as the root KMS for supporting SSE. These steps assume the AWS components meet the prerequisites.

Prior to starting these steps, create the following folders:

mkdir -P ~/kes/certs ~/kes/config

1) Download the MinIO Key Encryption Service

You can download the KES binary for running in baremetal environments, or use the KES container image for running in an orchestrated environment:

Download the latest stable release (0.16.1) of KES from github.com/minio/kes.

Select the binary appropriate for the host OS architecture. For example, hosts running X86-64 (Intel/AMD64) should download the kes-linux-amd64 package.

The following example code downloads the latest Linux AMD64-compatible binary and moves it to the system PATH:

wget https://github.com/minio/kes/releases/download/v0.16.1/kes-linux-amd64 -O /tmp/kes && \
chmod +x /tmp/kes && \
sudo mv /tmp/kes /usr/local/bin

kes --version

The following command uses Podman to download the latest stable KES (0.16.1) as a container image:

podman pull quay.io/minio/kes/v0.16.1

You can validate the container downloaded correctly by running the following command:

podman run kes --version

The output should reflect 0.16.1.

2) Generate the TLS Private and Public Key for KES

This step creates a self-signed TLS certificate for use with KES in evaluation or early development environments. The certificate expires within 30 days of creation.

For production environments, use certificates signed by a trusted Certificate Authority (CA). DO NOT use certificates generated using these instructions in production environments.

The following command creates the self-signed private and public key files using the kes tool identity new command:

kes tool identity new --server \
                      --key  ~/kes/certs/server.key  \
                      --cert ~/kes/certs/server.cert \
                      --ip   "127.0.0.1"             \
                      --dns  localhost

The following command creates the self-signed private and public key files using the kes tool identity new command. podman run --rm automatically removes the container when the command exists

podman run --rm -v ~/kes/certs:/data/certs                      \
           kes tool identity new --server                       \
                                 --key  /data/certs/server.key  \
                                 --cert /data/certs/server.cert \
                                 --ip   "127.0.0.1"             \
                                 --dns  localhost

This command outputs the keys to the ~/kes/certs directory on the host operating system.

3) Generate the TLS Private and Public Key for MinIO

KES uses mTLS for authorizing a connecting client to perform a requested cryptographic operation. This step creates a new TLS identity for the MinIO deployment to use in performing secure cryptographic operations on KES. The certificate expires within 30 days of creation.

For production environments, use certificates signed by a trusted Certificate Authority (CA). DO NOT use certificates generated using these instructions in production environments.

The following command creates the self-signed private and public key files using the kes tool identity new command:

kes tool identity new --server \
                      --key  ~/kes/certs/minio-kes.key  \
                      --cert ~/kes/certs/minio-kes.cert \
                      --ip   "127.0.0.1"             \
                      --dns  localhost

The command outputs the keys to the ~/kes/certs directory.

Use the kes tool identity of command to compute the identity hash for the certificate. This hash is required for configuring access to the KES server in a later step:

kes tool identify of ~/kes/certs/minio-kes.cert

The following command creates the self-signed private and public key files using the kes tool identity new command. podman run --rm automatically removes the container when the command exists

podman run --rm -v ~/kes/certs:/data/certs                     \
       kes tool identity new --key  /data/certs/minio-kes.key  \
                             --cert /data/certs/minio-kes.cert

This command outputs the keys to the ~/kes/certs directory on the host operating system.

Use the kes tool identity of command to compute the identity hash for the certificate. This hash is required for configuring access to the KES server in a later step:

sudo podman run --rm --v ~/kes/certs:/data/certs                \
                kes tool identity of /data/certs/minio-kes.cert

4) Create the KES Configuration File

KES uses a YAML-formatted configuration file. The following example YAML specifies the minimum required fields for enabling SSE using AWS Secrets Manager:

address: 0.0.0.0:7373

# Disable the root identity, as we do not need that level of access for
# supporting SSE operations.
root: disabled

# Specify the TLS keys generated in the previous step here
# For production environments, use keys signed by a known and trusted
# Certificate Authority (CA).
tls:
  key:  /data/certs/server.key
  cert: /data/certs/server.cert

# Create a policy named 'minio' that grants access to the
# /create, /generate, and /decrypt KES APIs for any key name
# KES uses mTLS to grant access to this policy, where only the client
# whose TLS certificate hash matches one of the "identities" can
# use this policy. Specify the hash of the MinIO server TLS certificate
# hash here.
policy:
  minio:
    allow:
    - /v1/key/create/*
    - /v1/key/generate/*
    - /v1/key/decrypt/*
    identities:
    - ${MINIO_IDENTITY_HASH} # Replace with the output of 'kes tool identity of minio-kes.cert'

# Specify the connection information for the KMS and Secrets Manager endpoint.
# The endpoint should be resolvable from the host.
# This example assumes that the associated AWS account has the necessary
# access key and secret key
keystore:
  secretsmanager:
    endpoint: secretsmanager.REGION.amazonaws # use the Secrets Manager endpoint for your region
    region: REGION # e.g. us-east-1
    kmskey: "" # Optional. The root AWS KMS key to use for cryptographic operations. Formerly described as the "Customer Master Key".
    credentials:
      accesskey: "${AWSACCESSKEY}" # AWS Access Key
      secretkey: "${AWSSECRETKEY}" # AWS Secret Key

Save the configuration file as ~/kes/config/kes-config.yaml. Any field with value ${VARIABLE} uses the environment variable with matching name as the value. You can use this functionality to set credentials without writing them to the configuration file.

  • Set MINIO_IDENTITY_HASH to the output of kes tool identity of minio-kes.cert.

  • Replace the REGION with the appropriate region for AWS Secrets Manager. The value must match for both endpoint and region.

  • Set AWSACCESSKEY and AWSSECRETKEY to the appropriate AWS Credentials.

5) Start KES

The first command allows KES to use the mlock system call without running as root. mlock ensures the OS does not write in-memory data to disk (swap memory) and mitigates the risk of cryptographic operations being written to unsecured disk at any time.

The second command starts the KES server in the foreground using the configuration file created in the last step. The --auth=off disables strict validation of client TLS certificates and is required if either the MinIO client or the root KMS server uses self-signed certificates.

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))

kes server --mlock                            \
           --config=~/kes/config/server-config.yaml  \
           --auth=off

KES listens on port 7373 by default. You can monitor the server logs from the terminal session. If you run KES without tying it to the current shell session (e.g. with nohup), use that methods associated logging system (e.g. nohup.txt).

The following command starts the KES server using the configuration file created in the last step. The command includes the necessary extensions that allow KES to use the mlock system call without running as root. mlock ensures the OS does not write in-memory data to disk (swap memory) and mitigates the risk of cryptographic operations being written to unsecured disk at any time.

podman run --rm -idt --cap-add=IPC_LOCK                         \
           --name kes-server                                    \
           -v ~/kes/certs:/data/certs                           \
           -v ~/kes/config:/data/config                         \
           -p 7373:7373                                         \
           kes server --mlock                                   \
                      --config=/data/config/server-config.yaml  \
                      --auth=off

The container starts using the specified configuration file and begins listening for client connections at por 7373. The server attempts to connect to the root KMS deployment specified in the server configuration file.

6) Generate a Cryptographic Key

MinIO requires that the EK exist on the root KMS before performing SSE operations using that key. Use kes key create or mc admin kms key create to create a new EK for use with SSE.

The following command uses the kes key create command to create a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.

export KES_SERVER=https://127.0.0.1:7373
export KES_CLIENT_KEY=~/kes/minio-kes.key
export KES_CLIENT_CERT=~/kes/minio-kes.cert

kes key create -k minio-backend-default-key

MinIO requires that the EK exist on the root KMS before performing SSE operations using that key. Use kes key create or mc admin kms key create to create a new EK for use with SSE.

The following command uses the kes key create command to create a new External Key (EK) stored on the root KMS server for use with encrypting the MinIO backend.

sudo podman exec -it kes-server /bin/bash

[root@ID /]# /kes key create -k                                      \
                             -e KES_SERVER=https://127.0.0.1:7373    \
                             -e KES_CLIENT_KEY=/data/minio-kes.key   \
                             -e KES_CLIENT_CERT=/data/minio-kes.cert \
                             minio-backend-default-key

7) Configure MinIO to connect to KES

Set the following environment variables to configure MinIO to connect to the KES server. Set these variables on all hosts running MinIO servers in the deployment. This command assumes the minio-kes.cert, minio-kes.key, and server.cert certificates are accessible at the specified location:

export MINIO_KMS_KES_ENDPOINT=https://HOSTNAME:7373
export MINIO_KMS_KES_CERT_FILE=~/minio-kes.cert
export MINIO_KMS_KES_KEY_FILE=~/minio-kes.key
export MINIO_KMS_KES_CAPATH=~/server.cert
export MINIO_KMS_KES_KEY_NAME=minio-backend-default-key

minio server [ARGUMENTS]
  • Replace HOSTNAME with the IP address or the hostname for the host machine running the KES server or pod started in the previous step.

  • Replace the minio server [ARGUMENTS] to match the command used to start the MinIO server on that host.

  • Add all other environment variables as required by the deployment.

MinIO uses the MINIO_KMS_KES_KEY_NAME key for the following cryptographic operations:

  • Encrypting the MinIO backend (IAM, configuration, etc.)

  • Performing SSE-KMS if the request does not include a specific EK.

  • Performing SSE-S3.

8) Enable Automatic Server-Side Encryption

The following command enables SSE-KMS on all objects written to the specified bucket:

mc mb ALIAS/encryptedbucket
mc encrypt set SSE-KMS encrypted-bucket-key ALIAS/encryptedbucket

Replace ALIAS with the alias of the MinIO deployment configured in the previous step.

Write a file to the bucket using mc cp or any S3-compatible SDK with a PutObject function. You can then run mc stat on the file to confirm the associated encryption metadata.

The following command enables SSE-S3 on all objects written to the specified bucket. MinIO uses the MINIO_KMS_KES_KEY_NAME key for performing SSE.

mc mb ALIAS/encryptedbucket
mc encrypt set SSE-S3 ALIAS/encryptedbucket

Replace ALIAS with the alias of the MinIO deployment configured in the previous step.

Write a file to the bucket using mc cp or any S3-compatible SDK with a PutObject function. You can then run mc stat on the file to confirm the associated encryption metadata.

Configuration Reference for AWS Root KMS

The following section describes each of the Key Encryption Service (KES) configuration settings for using AWS Secrets Manager and AWS KMS as the root Key Management Service (KMS) for SSE:

The following YAML describes the minimum required fields for configuring AWS Secrets Manager as an external KMS for supporting SSE.

Any field with value ${VARIABLE} uses the environment variable with matching name as the value. You can use this functionality to set credentials without writing them to the configuration file.

address: 0.0.0.0:7373
root: ${ROOT_IDENTITY}

tls:
  key: kes-server.key
  cert: kes-server.cert

policy:
  minio-server:
    allow:
      - /v1/key/create/*
      - /v1/key/generate/*
      - /v1/key/decrypt/*
    identities:
    - ${MINIO_IDENTITY}

keys:
  - name: "minio-encryption-key-alpha"
  - name: "minio-encryption-key-baker"
  - name: "minio-encryption-key-charlie"

keystore:
  secretsmanager:
    endpoint: secretsmanager.REGION.amazonaws
    region: REGION
    kmskey: ""
    credentials:
      accesskey: "${AWS_ACCESS_KEY}"
      secretkey: "${AWS_SECRET_KEY}"

Key

Description

address

The network address and port on which the KES server listens to on startup. Defaults to port 7373 on all host network interfaces.

root

The identity for the KES superuser (root) identity. Clients connecting with a TLS certificate whose hash (kes tool identity of client.cert) matches this value have access to all KES API operations.

You can specify 'disabled' to disable this identity and limit access based on the policy configuration.

tls

The TLS private key and certificate used by KES for establishing TLS-secured communications. Specify the full path to both the private .key and public .cert to the key and cert fields respectively.

policy

Specify one or more policies to control access to the KES server.

MinIO SSE requires access to only the following KES cryptographic APIs:

  • /v1/key/create/*

  • /v1/key/generate/*

  • /v1/key/decrypt/*

You can restrict the range of key names MinIO can create as part of performing SSE by specifying a prefix to replace the *. For example, minio-sse-* only grants access to create, generate, or decrypt keys using that prefix.

KES uses mTLS to authorize connecting clients by comparing the hash of the TLS certificate against the identities of each configured policy. Use the kes tool identity of command to compute the identity of the MinIO mTLS certificate and add it to the policy.<NAME>.identities array to associate MinIO to the <NAME> policy.

keys

Specify an array of keys which must exist on the root KMS for KES to successfully start. KES attempts to create the keys if they do not exist and exits with an error if it fails to create any key. KES does not accept any client requests until it completes validation of all specified keys.

keystore.secretsmanager

The configuration for the AWS Secrets Manager and AWS KMS.

  • endpoint - The endpoint for the Secrets Manager service, including the region.

  • approle - The AWS region to use for other AWS services.

  • kmskey - The root KMS Key to use for cryptographic operations. Formerly known as the Customer Master Key.

  • credentials - The AWS Credentials to use for performing authenticated operations against Secrets Manager and KMS.

    The specified credentials must have the appropriate permissions