Transition Objects from MinIO to Azure

The procedure on this page creates a new object lifecycle management rule that transition objects from a MinIO bucket to a remote storage tier on the Azure storage backend. This procedure supports use cases like moving aged data to low-cost public cloud storage solutions after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required Azure Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the Azure credentials provide read, write, list, and delete permissions for the remote bucket.

Refer to the Azure RBAC documentation for more complete guidance on configuring the required permissions.

Considerations

Exclusive Access to Remote Data

MinIO requires exclusive access to the transitioned data on the remote storage tier. MinIO ignores any objects in the remote bucket or bucket prefix not explicitly managed by the MinIO deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

  • No external mutation, migration, or deletion of objects on the remote storage.

  • No lifecycle management rules (e.g. transition or expiration) on the remote storage bucket.

MinIO stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. MinIO supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This tutorial includes the necessary syntax for setting this prefix.

Important

MinIO does not support changing the account name associated to an Azure remote tier. Azure storage backends are tied to the account, such that changing the account would change the storage backend and prevent access to any objects transitioned to the original account/backend.

Please contact MinIO Support if you need situation-specific guidance around configuring Azure remote tiers.

Availability of Remote Data

MinIO creates metadata for each transitioned object that identifies its location on the remote storage. This metadata is required for accessing the object, such that applications cannot access a transition object independent of MinIO. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the MinIO deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require BC/DR protections should implement MinIO Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the MinIO deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the MinIO deployment. Replace this value with the appropriate alias for the MinIO deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

wget -O - https://docs.min.io/minio/baremetal/examples/LifecycleManagementAdmin.json | \
mc admin policy add Alpha LifecycleAdminPolicy /dev/stdin
mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
mc admin policy set Alpha LifecycleAdminPolicy user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and MinIO Policy Based Access Control for more complete documentation on MinIO users and policies respectively.

2) Configure the Remote Storage Tier

Use the mc admin tier add command to add a new remote storage tier:

mc admin tier add --azure TARGET TIER_NAME \
   --endpoint https://HOSTNAME
   --bucket BUCKET \
   --prefix PREFIX
   --account-name ACCOUNT \
   --account-key KEY \
   --region REGION

The example above uses the following arguments:

Argument

Description

TARGET

The alias of the MinIO deployment on which to configure the remote tier.

TIER_NAME

The name to associate with the new Azure blob remote storage tier. Specify the name in all-caps, e.g. AZURE_TIER. This value is required in the next step.

HOSTNAME

The URL endpoint for the Azure storage backend.

BUCKET

The name of the bucket on the Azure storage backend to which MinIO transitions objects.

PREFIX

The optional bucket prefix within which MinIO transitions objects.

MinIO stores all transitioned objects in the specified BUCKET under a unique per-deployment prefix value. Omit this argument to use only that value for isolating and organizing data within the remote storage.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This prefix should provide a clear reference back to the source MinIO deployment to faciliate ease of operations related to diagnostics, maintenance, or disaster recovery.

ACCOUNT

The account name MinIO uses to access the bucket. The account name must correspond to an Azure user with the required permissions.

You cannot change this account name after creating the tier.

KEY

The corresponding key for the specified ACCOUNT.

REGION

The Azure blob storage region of the specified BUCKET. You can safely omit this option if the HOSTNAME includes the region.

3) Create and Apply the Transition Rule

Use the mc ilm add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

mc ilm add ALIAS/BUCKET \
--storage-class TIERNAME \
--transition-days DAYS \
--noncurrentversion-transition-days NONCURRENT_DAYS
--noncurrentversion-transition-storage-class TIERNAME

The example above specifies the following arguments:

Argument

Description

ALIAS

Specify the alias of the MinIO deployment for which you are creating the lifecycle management rule.

BUCKET

Specify the full path to the bucket for which you are creating the lifecycle management rule.

TIERNAME

The remote storage tier to which MinIO transitions objects. Specify the remote storage tier name created in the previous step.

If you want to transition noncurrent object versions to a distinct remote tier, specify a different tier name for --noncurrentversion-transition-storage-class.

DAYS

The number of calendar days after which MinIO marks an object as eligible for transition. Specify the number of days as an integer, e.g. 30 for 30 days.

NONCURRENT_DAYS

The number of calendar days after which MinIO marks a noncurrent object version as eligible for transition. MinIO specifically measures the time since an object became non-current instead of the object creation time. Specify the number of days as an integer, e.g. 90 for 90 days.

Omit this value to ignore noncurrent object versions.

This option has no effect on non-versioned buckets.

4) Verify the Transition Rule

Use the mc ilm ls command to review the configured transition rules:

mc ilm ls ALIAS/PATH --transition
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.