Welcome to the upcoming version of the MinIO Documentation! The content on this page is under active development and may change at any time. If you can't find what you're looking for, check our legacy documentation. Thank you for your patience.

Transition Objects from MinIO to S3

The procedure on this page creates a new object lifecycle management rule that transition objects from a MinIO bucket to a remote storage tier on the Amazon Web Services S3 storage backend or an S3-compatible service. This procedure supports use cases such as tiering objects to low-cost or archival storage after a certain time period or calendar date.

Requirements

Install and Configure mc

This procedure uses mc for performing operations on the MinIO cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias command to create an alias for the source MinIO cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required MinIO Permissions

MinIO requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

MinIO also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:.

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required S3 Permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, MinIO requires the remote tier credentials provide read, write, list, and delete permissions for the remote bucket.

For example, the following policy provides the necessary permission for transitioning objects into and out of the remote tier:

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket"
            ],
            "Sid": ""
      },
      {
            "Action": [
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket/*"
            ],
            "Sid": ""
      }
   ]
}

Modify the Resource for the bucket into which MinIO tiers objects.

Refer to the Amazon S3 Permissions documentation for more complete guidance on configuring the required permissions.

Considerations

Lifecycle Management Object Scanner

MinIO uses a scanner process to check objects against all configured lifecycle management rules. Slow scanning due to high IO workloads or limited system resources may delay application of lifecycle management rules. See Lifecycle Management Object Scanner for more information.

Exclusive Bucket Access

MinIO retains the minimum object metadata required to support retrieving objects transitioned to a remote tier. MinIO therefore requires exclusive access to the data on the remote storage tier. Object retrieval assumes no external mutation, migration, or deletion of stored objects.

MinIO also ignores any objects in the remote bucket or bucket prefix not explicitly managed by MinIO.

Procedure

1) Configure User Accounts and Policies for Lifecycle Management

This step creates users and policies on the MinIO deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

The following example uses Alpha as a placeholder alias for the MinIO deployment. Replace this value with the appropriate alias for the MinIO deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

wget -O - https://docs.min.io/minio/baremetal/examples/LifecycleManagementAdmin.json | \
mc admin policy add Alpha LifecycleAdminPolicy /dev/stdin
mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
mc admin policy set Alpha LifecycleAdminPolicy user=alphaLifecycleAdmin

This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and MinIO Policy Based Access Control for more complete documentation on MinIO users and policies respectively.

2) Configure the Remote Storage Tier

Use the mc admin tier add command to add an Amazon S3 service as the new remote storage tier:

mc admin tier add s3 TARGET TIER_NAME \
   --endpoint https://HOSTNAME \
   --bucket BUCKET \
   --prefix PREFIX
   --access-key ACCESS_KEY \
   --secret-key SECRET_KEY \
   --region REGION \
   --storage-class STORAGE_CLASS

The example above uses the following arguments:

Argument

Description

TARGET

The alias of the MinIO deployment on which to configure the S3 remote tier.

TIER_NAME

The name to associate with the new S3 remote storage tier. This value is required in the next step.

HOSTNAME

The URL endpoint for the S3 storage backend.

BUCKET

The name of the bucket on the S3 storage backend to which MinIO transitions objects.

PREFIX

The optional bucket prefix within which MinIO transitions objects. Omit this argument to transition objects to the bucket root.

ACCESS_KEY

The S3 access key MinIO uses to access the bucket. The access key must correspond to an IAM user with the required permissions.

SECRET_KEY

The corresponding secret key for the specified ACCESS_KEY.

REGION

The AWS S3 region of the specified BUCKET. You can safely omit this option if the HOSTNAME includes the region.

STORAGE_CLASS

The S3 storage class to which MinIO transitions objects. Specify one of the following supported storage classes:

  • STANDARD

  • REDUCED

3) Create and Apply the Transition Rule

Use the mc ilm add command to create a new transition rule for the bucket. The following tabs contain examples for transitioning objects on a calendar date or after a number of calendar days.

mc ilm add ALIAS/BUCKET \
--storage-class TIERNAME \
--transition-date DATE \
--noncurrentversion-transition-date NONCURRENT_DAYS

The example above specifies the following arguments:

Argument

Description

ALIAS

Specify the alias of the MinIO deployment for which you are creating the lifecycle management rule.

BUCKET

Specify the full path to the bucket for which you are creating the lifecycle management rule.

TIERNAME

The remote storage tier to which MinIO transitions objects. Specify the remote storage tier name created in the previous step.

DATE

The ISO-8601-formatted calendar date after which MinIO marks an object as eligible for transition.

NONCURRENT_DAYS

The number of calendar days after which MinIO marks a noncurrent object version as eligible for transition. Omit this value to ignore noncurrent object versions.

This option has no effect on non-versioned buckets.

mc ilm add ALIAS/BUCKET \
--storage-class TIERNAME \
--transition-days DAYS \
--noncurrentversion-transition-days NONCURRENT_DAYS

The example above specifies the following arguments:

Argument

Description

ALIAS

Specify the alias of the MinIO deployment for which you are creating the lifecycle management rule.

BUCKET

Specify the full path to the bucket for which you are creating the lifecycle management rule.

TIERNAME

The remote storage tier to which MinIO transitions objects. Specify the remote storage tier name created in the previous step.

DAYS

The number of calendar days after which MinIO marks an object as eligible for transition.

NONCURRENT_DAYS

The number of calendar days after which MinIO marks a noncurrent object version as eligible for transition. Omit this value to ignore noncurrent object versions.

This option has no effect on non-versioned buckets.

3) Verify the Transition Rule

Use the mc ilm list command to review the configured transition rules:

mc ilm list ALIAS/PATH --transition
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.