Transition objects from AIStor to a remote AIStor deployment

The procedure on this page creates a new object lifecycle management rule that transitions objects from a bucket on a primary AIStor deployment to a bucket on a remote AIStor deployment. This procedure supports cost-management strategies such as tiering objects from a “hot” AIStor deployment using NVMe storage to a “warm” AIStor deployment using SSD.

Requirements

Install and configure mc

This procedure uses mc for performing operations on the AIStor cluster. Install mc on a machine with network access to both source and destination clusters. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias set command to create an alias for the source AIStor cluster. Alias creation requires specifying an access key for a user on the source and destination clusters. The specified users must have permissions for configuring and applying transition operations.

Required source AIStor permissions

AIStor requires the following permissions scoped to the bucket or buckets for which you are creating lifecycle management rules.

AIStor also requires the following administrative permissions on the cluster in which you are creating remote tiers for object transition lifecycle management rules:

For example, the following policy provides permission for configuring object transition lifecycle management rules on any bucket in the cluster:

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "admin:SetTier",
               "admin:ListTier"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteTierManagement"
      },
      {
            "Action": [
               "s3:PutLifecycleConfiguration",
               "s3:GetLifecycleConfiguration"
            ],
            "Resource": [
                        "arn:aws:s3:::*"
            ],
            "Effect": "Allow",
            "Sid": "EnableLifecycleManagementRules"
      }
   ]
}

Required remote AIStor permissions

Object transition lifecycle management rules require additional permissions on the remote storage tier. Specifically, AIStor requires the remote tier credentials provide read, write, list, and delete permissions for the remote bucket.

For example, the following policy on the remote AIStor deployment provides the necessary permission for transitioning objects into and out of the remote tier:

{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Action": [
               "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket"
            ],
            "Sid": ""
      },
      {
            "Action": [
               "s3:GetObject",
               "s3:PutObject",
               "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
               "arn:aws:s3:::MyDestinationBucket/*"
            ],
            "Sid": ""
      }
   ]
}

Modify the Resource for the bucket into which AIStor tiers objects.

Refer to the Access Management documentation for more complete guidance on configuring the required permissions.

Remote bucket must exist

Create the remote bucket prior to configuring lifecycle management tiers or rules using that bucket as the target.

If the remote bucket contains existing data, use the prefix feature to isolate transitioned objects from any other objects on that bucket.

Considerations

Lifecycle management object scanner

AIStor uses a scanner process to check objects against all configured lifecycle management rules. Slow scanning due to high IO workloads or limited system resources may delay application of lifecycle management rules.

Exclusive access to remote data

AIStor requires exclusive access to the transitioned data on the remote storage tier. Object metadata on the “hot” AIStor source is strongly linked to the object data on the “warm/cold” remote tier. AIStor cannot retrieve object data without access to the remote, nor can the remote be used to restore lost metadata on the source.

All access to the transitioned objects must occur through AIStor using S3 API operations only. Manually modifying a transitioned object - whether the metadata on the “hot” AIStor tier or the object data on the remote “warm/cold” tier - may result in loss of that object data.

AIStor ignores any objects in the remote bucket or bucket prefix not explicitly managed by the AIStor deployment. Automatic transition and transparent object retrieval depend on the following assumptions:

  • No external mutation, migration, or deletion of objects on the remote storage.
  • No lifecycle management rules (for example, transition or expiration) on the remote storage bucket.

AIStor stores all transitioned objects in the remote storage bucket or resource under a unique per-deployment prefix value. This value is not intended to support identifying the source deployment from the backend. AIStor supports an additional optional human-readable prefix when configuring the remote target, which may facilitate operations related to diagnostics, maintenance, or disaster recovery.

MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other AIStor deployments. This tutorial includes the necessary syntax for setting this prefix.

Availability of remote data

AIStor tiering behavior depends on the remote storage returning objects immediately (milliseconds to seconds) upon request. AIStor therefore cannot support remote storage which requires rehydration, wait periods, or manual intervention.

AIStor creates metadata for each transitioned object that identifies its location on the remote storage. Applications cannot trivially identify and access a transitioned object independent of AIStor. Availability of the transitioned data therefore depends on the same core protections that erasure coding and distributed deployment topologies provide for all objects on the AIStor deployment. Using object transition does not provide any additional business continuity or disaster recovery benefits.

Workloads that require protections should implement AIStor Server-Side replication. Replication ensures objects remains preserved on the remote replication site, such that you can resynchronize from the remote in the event of partial or total data loss. See Resynchronization (Disaster Recovery) for more complete documentation on using replication to recover after partial or total data loss.

Procedure

  1. Configure user accounts and policies for lifecycle management

    This step creates users and policies on the AIStor deployment for supporting lifecycle management operations. You can skip this step if the deployment already has users with the necessary permissions.

    The following example uses Alpha as a placeholder alias for the AIStor deployment. Replace this value with the appropriate alias for the AIStor deployment on which you are configuring lifecycle management rules. Replace the password LongRandomSecretKey with a long, random, and secure secret key as per your organizations best practices for password generation.

    wget -O - https://min.io/docs/minio/linux/examples/LifecycleManagementAdmin.json | \
    mc admin policy create Alpha LifecycleAdminPolicy /dev/stdin
    mc admin user add Alpha alphaLifecycleAdmin LongRandomSecretKey
    mc admin policy attach Alpha LifecycleAdminPolicy --user=alphaLifecycleAdmin
    

    This example assumes that the specified aliases have the necessary permissions for creating policies and users on the deployment. See User Management and AIStor Policy Based Access Control for more complete documentation on AIStor users and policies respectively.

  2. Configure the remote storage tier

    Use the mc ilm tier add command to add the remote AIStor deployment as the new remote storage tier:

    mc ilm tier add minio TARGET TIER_NAME  \
       --endpoint https://HOSTNAME       \
       --access-key ACCESS_KEY           \
       --secret-key SECRET_KEY           \
       --bucket BUCKET                   \
       --prefix PREFIX                   \
       --storage-class STORAGE_CLASS     \
       --region REGION
    

    The example above uses the following arguments:

    Argument Description
    ALIAS The alias of the AIStor deployment on which to configure the AIStor remote tier.
    TIER_NAME The name to associate with the new AIStor remote storage tier. Specify the name in all-caps, e.g. MINIO_WARM_TIER. This value is required in the next step.
    HOSTNAME The URL endpoint for the AIStor storage backend.
    ACCESS_KEY The access key AIStor uses to access the bucket. The access key must correspond to an IAM user with the required permissions.
    SECRET_KEY The corresponding secret key for the specified ACCESS_KEY.
    BUCKET The name of the bucket on the remote AIStor deployment to which the SOURCE transitions objects.
    PREFIX The optional bucket prefix within which AIStor transitions objects.

    AIStor stores all transitioned objects in the specified BUCKET under a unique per-deployment prefix value. Omit this argument to use only that value for isolating and organizing data within the remote storage.

    MinIO recommends specifying this optional prefix for remote storage tiers which contain other data, including transitioned objects from other MinIO deployments. This prefix should provide a clear reference back to the source AIStor deployment to facilitate ease of operations related to diagnostics, maintenance, or disaster recovery.
    STORAGE_CLASS The Erasure Coding storage class AIStor applies to objects transitions to the remote AIStor bucket.
    Specify one of the following supported storage classes:

    - STANDARD Recommended
    - REDUCED
    REGION The AIStor region of the specified BUCKET.

    AIStor deployments typically do not require setting a region as part of setup.
    Only include this option if you explicitly set the MINIO_SITE_REGION configuration setting for the deployment.
  3. Create and apply the transition rule

    Use the mc ilm rule add command to create a new transition rule for the bucket. The following example configures transition after the specified number of calendar days:

    mc ilm rule add ALIAS/BUCKET \
    --transition-tier TIERNAME \
    --transition-days DAYS \
    --noncurrent-transition-days NONCURRENT_DAYS
    --noncurrent-transition-tier TIERNAME
    

    The example above specifies the following arguments:

    Argument Description
    ALIAS Specify the alias of the AIStor deployment for which you are creating the lifecycle management rule.
    BUCKET Specify the full path to the bucket for which you are creating the lifecycle management rule.
    TIERNAME The remote storage tier to which AIStor transitions objects.
    Specify the remote storage tier name created in the previous step.

    If you want to transition noncurrent object versions to a distinct remote tier, specify a different tier name for
    --noncurrent-transition-tier.
    DAYS The number of calendar days after which AIStor marks an object as eligible for transition. Specify the number of days as an integer, for example 30 for 30 days.
    NONCURRENT_DAYS The number of calendar days after which AIStor marks a noncurrent object version as eligible for transition. AIStor specifically measures the time since an object became non-current instead of the object creation time. Specify the number of days as an integer, for example 90 for 90 days.

    Omit this value to ignore noncurrent object versions.

    This option has no effect on non-versioned buckets.
  4. Verify the transition rule

    Use the mc ilm rule ls command to review the configured transition rules:

    mc ilm rule ls ALIAS/PATH --transition
    
    • Replace ALIAS with the alias of the AIStor deployment.
    • Replace PATH with the name of the bucket for which to retrieve the configured lifecycle management rules.
All rights reserved 2024-Present, MinIO, Inc.