Enable Multi-Site Server-Side Bucket Replication

The procedure on this page configures automatic server-side bucket replication between multiple MinIO deployments. Multi-Site Active-Active replication builds on the Enable Two-Way Server-Side Bucket Replication procedure with additional considerations required to ensure predictable replication behavior across all sites.

Active-Active Replication synchronizes data between multiple remote deployments.

Multi-Site Active-Active replication configurations can span multiple racks, datacenters, or geographic locations. Complexity of configuring and maintaining multi-site configurations generally increase with the number of sites and size of each site. Enterprises looking to implement multi-site replication should consider leveraging MinIO SUBNET support to access the expertise, planning, and engineering resources required for addressing that use case.

See also

Requirements

Install and Configure mc with Access to Both Clusters.

This procedure uses mc for performing operations on both the source and destination MinIO cluster. Install mc on a machine with network access to both source and destination deployments. See the mc Installation Quickstart for instructions on downloading and installing mc.

Use the mc alias command to create an alias for both MinIO deployments. Alias creation requires specifying an access key for a user on the cluster. This user must have permission to create and manage users and policies on the cluster. Specifically, ensure the user has at minimum:

Required Permissions

Bucket replication requires specific permissions on the source and destination deployments to configure and enable replication rules.

The following policy provides permissions for configuring and enabling replication on a deployment.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}
  • The "EnableRemoteBucketConfiguration" statement grants permission for creating a remote target for supporting replication.

  • The "EnableReplicationRuleConfiguration" statement grants permission for creating replication rules on a bucket. The "arn:aws:s3:::* resource applies the replication permissions to any bucket on the source deployment. You can restrict the user policy to specific buckets as-needed.

The following code creates a MinIO-managed user with the necessary policy. Replace the TARGET with the alias of the MinIO deployment on which you are configuring replication:

wget -O - https://docs.min.io/minio/baremetal/examples/ReplicationAdminPolicy.json | \
mc admin policy add TARGET ReplicationAdminPolicy /dev/stdin
mc admin user add TARGET ReplicationAdmin LongRandomSecretKey
mc admin policy set TARGET ReplicationAdminPolicy user=ReplicationAdmin

MinIO deployments configured for Active Directory/LDAP or OpenID Connect user management should instead create a dedicated service account for bucket replication.

The following policy provides permissions for enabling synchronization of replicated data into the deployment.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}
  • The "EnableReplicationOnBucket" statement grants permission for a remote target to retrieve bucket-level configuration for supporting replication operations on all buckets in the MinIO deployment. To restrict the policy to specific buckets, specify those buckets as an element in the Resource array similar to "arn:aws:s3:::bucketName".

  • The "EnableReplicatingDataIntoBucket" statement grants permission for a remote target to synchronize data into any bucket in the MinIO deployment. To restrict the policy to specific buckets, specify those buckets as an element in the Resource array similar to "arn:aws:s3:::bucketName/*".

The following code creates a MinIO-managed user with the necessary policy. Replace the TARGET with the alias of the MinIO deployment on which you are configuring replication:

wget -O - https://docs.min.io/minio/baremetal/examples/ReplicationRemoteUserPolicy.json | \
mc admin policy add TARGET ReplicationRemoteUserPolicy /dev/stdin
mc admin user add TARGET ReplicationRemoteUser LongRandomSecretKey
mc admin policy set TARGET ReplicationRemoteUserPolicy user=ReplicationRemoteUser

MinIO deployments configured for Active Directory/LDAP or OpenID Connect user management should instead create a dedicated service account for bucket replication.

See mc admin user, mc admin user svcacct, and mc admin policy for more complete documentation on adding users, service accounts, and policies to a MinIO deployment.

Replication Requires Matching Object Encryption Settings

MinIO supports replication of objects encrypted using SSE-KMS and SSE-S3:

  • For objects encrypted using SSE-KMS, MinIO requires that the target bucket support SSE-KMS encryption of objects using the same key names used to encrypt objects on the source bucket.

  • For objects encrypted using SSE-S3, MinIO requires that the target bucket also support SSE-S3 encryption of objects regardless of key name.

As part of the replication process, MinIO decrypts the object on the source bucket and transmits the unencrypted object over the network. The destination MinIO deployment then re-encrypts the object using the encryption settings from the target. MinIO therefore strongly recommends enabling TLS on both source and destination deployments to ensure the safety of objects during transmission.

MinIO does not support replicating client-side encrypted objects (SSE-C).

Replication Requires MinIO Deployments

MinIO server-side replication only works between MinIO deployments. Both the source and destination deployments must run MinIO.

To configure replication between arbitrary S3-compatible services, use mc mirror.

Replication Requires Versioning

MinIO relies on the immutability protections provided by versioning to support replication and resynchronization.

Use mc version info to validate the versioning status of both the healthy source and unhealthy target buckets. Use the mc version enable command to enable versioning as necessary.

Replication Requires Matching Object Locking State

MinIO supports replicating objects held under WORM Locking. Both replication buckets must have object locking enabled for MinIO to replicate the locked object. For active-active configuration, MinIO recommends using the same retention rules on both buckets to ensure consistent behavior across sites.

You must enable object locking during bucket creation as per S3 behavior. You can then configure object retention rules at any time. Configure the necessary rules on the unhealthy target bucket prior to beginning this procedure.

Considerations

Use Consistent Replication Settings

MinIO supports customizing the replication configuration to enable or disable the following replication behaviors:

  • Replication of delete operations

  • Replication of delete markers

  • Replication of existing objects

  • Replication of metadata-only changes

When configuring replication rules for a bucket, ensure that all MinIO deployments participating in multi-site replication use the same replication behaviors to ensure consistent and predictable synchronization of objects.

Replication of Existing Objects

MinIO supports automatically replicating existing objects in a bucket.

MinIO requires explicitly enabling replication of existing objects using the mc replicate add --replicate or mc replicate edit --replicate and including the existing-objects replication feature flag. This procedure includes the required flags for enabling replication of existing objects.

Replication of Delete Operations

MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate versioning Delete Markers and the deletion of specific versioned objects:

  • For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.

  • For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.

MinIO requires explicitly enabling replication of delete operations using the mc replicate add --replicate or mc replicate edit --replicate. This procedure includes the required flags for enabling replication of delete operations and delete markers.

MinIO does not replicate delete operations resulting from the application of lifecycle management expiration rules. Configure matching expiration rules for the bucket on all replication sites to ensure consistent application of object expiration.

Procedure

This procedure requires repeating steps for each MinIO deployment participating in the multi-site replication configuration. Depending on the number of deployments, this procedure may require significant time and care in implementation. MinIO recommends reading through the procedure before attempting to implement the documented steps.

This procedure uses the placeholder ALIAS to reference the alias each MinIO deployment being configured for replication. Replace these values with the appropriate alias for each MinIO deployment.

This procedure assumes each alias corresponds to a user with the necessary replication permissions.

1) Create the Replication Remote Target

Use the mc admin bucket remote add command to create a replication target for the each deployment. MinIO supports one remote target per destination bucket. You cannot create multiple remote targets for the same destination bucket.

mc admin bucket remote add ALIAS/BUCKET \
   https://ReplicationRemoteUser:LongRandomSecretKey@HOSTNAME/BUCKET \
   --service "replication"
  • Replace BUCKET with the name of the bucket on the ALIAS deployment to use as the replication source. Replace ALIAS with the alias of the MinIO deployment on which you are configuring replication.

  • Replace HOSTNAME with the URL of the remote MinIO deployment.

  • Replace BUCKET with the name of the bucket on the remote deployment to use as the replication destination.

The command returns an ARN similar to the following:

Role ARN = 'arn:minio:replication::<UUID>:BUCKET'

Copy the ARN string for use in the next step, noting the MinIO deployment on which it was created.

Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments Alpha, Baker, and Charlie would require repeating this step on each deployment for each remote. Specifically:

  • The Alpha deployment would perform this step once for Baker and once for Charlie.

  • The Baker deployment would perform this step once for Alpha and once for Charlie.

  • The Charlie deployment would perform this step once for Baker and once for Alpha.

2) Create a New Bucket Replication Rule

Use the mc replicate add command to add the new server-side replication rule to the each MinIO deployment.

mc replicate add ALIAS/BUCKET \
   --remote-bucket 'arn:minio:replication::<UUID>:BUCKET' \
   --replicate "delete,delete-marker,existing-objects"
  • Replace BUCKET with the name of the bucket on the ALIAS deployment to use as the replication source. Replace ALIAS with the alias of the MinIO deployment on which you are configuring replication.

  • Replace the --remote-bucket value with the ARN returned in the previous step. Ensure you specify the ARN created on the ALIAS deployment. You can use mc admin bucket remote ls to list all remote ARNs configured on the deployment.

  • The --replicate "delete,delete-marker,existing-objects" flag enables the following replication features:

    See mc replicate add --replicate for more complete documentation. Omit these fields to disable replication of delete operations or replication of existing objects respectively.

Specify any other supported optional arguments for mc replicate add.

Repeat these commands for each remote MinIO deployment participating in the multi-site replication configuration. For example, a multi-site replication configuration consisting of MinIO deployments Alpha, Baker, and Charlie would require repeating this step on each deployment for each remote. Specifically:

  • The Alpha deployment would perform this step once for Baker and once for Charlie.

  • The Baker deployment would perform this step once for Alpha and once for Charlie.

  • The Charlie deployment would perform this step once for Baker and once for Alpha.

3) Validate the Replication Configuration

Use mc cp to copy a new object the bucket on any of the deployments:

mc cp ~/foo.txt ALIAS/BUCKET

Use mc ls to verify the object exists on each remote deployment:

mc ls REMOTE/BUCKET

Repeat this test on each of the deployments by copying a new unique file and checking the other deployments for that file.

You can also use mc stat to check the file to check the current replication stage of the object.