Migrate Encryption Keys

This page documents migration of encryption keys from the legacy MinIO Key Encryption Service (KES) to AIStor Key Manager. Key Manager holds all cryptographic keys in an encrypted format without relying on an external keystore.

Once you complete this operation, you can switch your AIStor deployments to use Key Manager for supporting Server Side Encryption operations.

Migration of cryptographic keys is a critical operation that can lead to data loss if performed incorrectly. Before proceeding, you must do the following:

Halt creation of new encryption keys at the KES backend
Take a backup of encryption keys from the third-party Key Management Service (KMS)
Take a backup of the KES configuration
Verify that currently-encrypted data is readable

Consider opening a SUBNET ticket prior to beginning any key migration operation.

Prerequisites

This procedure assumes the following:

An existing Key Manager deployment
An existing Key Encryption Service and AIStor installation
An existing AIStor deployment with access to the mc client

This procedure uses the SysAdmin identity key for performing operations on the Key Manager deployment. Retrieve this key before starting the procedure or recompute it using the minkms identity command and the Key Manager configuration file:

minkms identity --config /opt/minkms/config.yaml

Use the k1:KEYSTRING API key as part of this procedure.

Procedure

Validate connectivity to KES and Key Manager

This procedure requires the local host have access to both deployments with the ability to validate TLS certificates. Install the necessary Certificate Authority certs including root and intermediate certificates to the host machine’s trust store. You can use the following commands to verify access to the deployments:
```
curl -X GET <KES-endpoint>/version
curl -X GET <KeyManager-endpoint>/version
```
Install the KES and Key Manager CLI tools
- Install KES
- Install Key Manager
Verify that each CLI can connect to it’s respective remote service. For example, the kes key ls and minkms env commands should both return valid outputs.
Create an enclave for the migrated keys

Key Manager uses enclaves to isolate cryptographic keys from other clients. Create a new enclave for storing the migrated keys. If you have an existing empty enclave for storing keys you can skip this step.
```
minkms add-enclave aistor-primary --api-key k1:KEYSTRING
```
Replace the enclave name aistor-primary with a name that corresponds to the appropriate AIStor deployment.
Create a new enclave admin

Key Manager uses identities to control access and permissions to enclaves. Create a new admin identity for the AIStor deployment:
```
minkms add-identity --admin --enclave aistor-primary --api-key k1:KEYSTRING
```
The command outputs the enclave admin identity and API key. Use the API key when reconfiguring the AIStor deployment to communicate with the Key Manager.
Retrieve the KES configuration file

The kes migrate command requires a copy of the KES configuration file to operate.
- For Linux installations, copy the configuration file from the remote KES host to your local machine.
- For Kubernetes installations, copy the configuration file from the YAML, secret, or other resource used to run KES.
Start the migration process

Validate that all key creation requests against KES have halted prior to starting this step.

Run the kes migrate command to begin the migration. The following example assumes a config file located in the user’s home directory as kes-config.yaml and a KES host located at the specified hostname and port.
```
kes migrate \
  --from ~/kes-config.yaml \
  --server 'https://kes-1.example.net:7373' \
  --enclave aistor-primary \
  --api-key k1:KEYSTRING
```
Replace the example values above to reflect your actual deployment topology.

The migration process returns the number of keys migrated once completed.
Verify migration

Use the kes key ls command to validate the number of keys currently managed on KES.

Use the minkms ls-key --enclave ENCLAVE command to validate the number of keys migrated to the Key Manager enclave.

If both values match, the migration succeeded for all keys stored on KES.

Migrate AIStor to Key Manager

AIStor uses the following environment variables for controlling access to a Key Manager deployment:

Environment Variable	Description
MINIO_KMS_SERVER	Specify a comma-separated list of Key Manager hostnames. Alternatively specify the load balancer, reverse proxy, or other network control plane component responsible for managing connections to the Key Manager hosts.
MINIO_KMS_API_KEY	Specify the `k1` prefixed enclave admin API key created in an earlier step.
MINIO_KMS_ENCLAVE	Specify the name of the enclave created in an earlier step.
MINIO_KMS_SSE_KEY	Specify the previous value of `MINIO_KMS_KES_KEY_NAME`.

Comment out all environment variables prefixed with MINIO_KMS_KES_.

For Linux deployments, edit the /etc/default/minio file on all nodes. Validate uniformity of environment file post-modification using a utility like shasum -a 256.
For Kubernetes, check the Helm values.yaml or Kubernetes resource file used to deploy AIStor to validate the deployed configuration. You may need to modify the configuration secret or the environment variables passed directly through the object description to make the necessary changes.

Once you have updated the configurations, you can restart the AIStor deployment using mc admin service restart. Monitor logs to ensure the service starts up as expected.

Validate AIStor functionality

Attempt to use mc head to test retrieval of encrypted objects on the AIStor deployment. All objects previously encrypted with KES should return without error as AIStor uses Key Manager to retrieve those keys.