Welcome to the upcoming version of the MinIO Documentation! The content on this page is under active development and may change at any time. If you can't find what you're looking for, check our legacy documentation. Thank you for your patience.

mc replicate

Description

The mc replicate command configures Server-Side Bucket Replication between MinIO deployments.

Server-Side Replication Requires MinIO Source and Destination

MinIO server-side replication only works between MinIO deployments. Both the source and destination deployments must run MinIO.

To configure replication between arbitrary S3-compatible services, use mc mirror.

Enable Versioning on Source and Destination Buckets

MinIO relies on the immutability protections provided by versioning to synchronize objects between the source and replication target.

Use the mc version enable command to enable versioning on both the source and destination bucket before starting this procedure:

mc version enable ALIAS/PATH
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the bucket on which to enable versioning.

Required Permissions

MinIO strongly recommends creating users specifically for supporting bucket replication operations. See mc admin user and mc admin policy for more complete documentation on adding users and policies to a MinIO deployment.

The following policy provides permissions for configuring and enabling replication on a deployment.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}
  • The "EnableRemoteBucketConfiguration" statement grants permission for creating a remote target for supporting replication.

  • The "EnableReplicationRuleConfiguration" statement grants permission for creating replication rules on a bucket. The "arn:aws:s3:::* resource applies the replication permissions to any bucket on the source deployment. You can restrict the user policy to specific buckets as-needed.

Use the mc admin policy add to add this policy to each deployment acting as a replication source. Use mc admin user add to create a user on the deployment and mc admin policy set to associate the policy to that new user.

The following policy provides permissions for enabling synchronization of replicated data into the deployment.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}
  • The "EnableReplicationOnBucket" statement grants permission for a remote target to retrieve bucket-level configuration for supporting replication operations on all buckets in the MinIO deployment. To restrict the policy to specific buckets, specify those buckets as an element in the Resource array similar to "arn:aws:s3:::bucketName".

  • The "EnableReplicatingDataIntoBucket" statement grants permission for a remote target to synchronize data into any bucket in the MinIO deployment. To restrict the policy to specific buckets, specify those buckets as an element in the Resource array similar to "arn:aws:s3:::bucketName/*".

Use the mc admin policy add to add this policy to each deployment acting as a replication target. Use mc admin user add to create a user on the deployment and mc admin policy set to associate the policy to that new user.

Replication of Existing Objects

Starting with mc RELEASE.2021-06-13T17-48-22Z and minio RELEASE.2021-06-07T21-40-51Z, MinIO supports automatically replicating existing objects in a bucket. MinIO existing object replication implements functionality similar to AWS: Replicating existing objects between S3 buckets without the overhead of contacting technical support.

  • To enable replication of existing objects when creating a new replication rule, include "existing-objects" to the list of replication features specified to mc replicate add --replicate.

  • To enable replication of existing objects for an existing replication rule, add "existing-objects" to the list of existing replication features using mc replicate add --replicate. You must specify all desired replication features when editing the replication rule.

See Replication of Existing Objects for more complete documentation on this behavior.

Synchronization of Metadata Changes

MinIO supports two-way active-active replication configurations, where MinIO synchronizes new and modified objects between a bucket on two MinIO deployments. Starting with mc RELEASE.2021-05-18T03-39-44Z, MinIO by default synchronizes metadata-only changes to a replicated object back to the “source” deployment. Prior to the this update, MinIO did not support synchronizing metadata-only changes to a replicated object.

With metadata synchronization enabled, MinIO resets the object replication status to indicate replication eligibility. Specifically, when an application performs a metadata-only update to an object with the REPLICA status, MinIO marks the object as PENDING and eligible for replication.

To disable metadata synchronization, use the mc replicate edit --replicate command and omit replica-metadata-sync from the replication feature list.

Replication of Delete Operations

MinIO supports replicating delete operations onto the target bucket. Specifically, MinIO can replicate both Delete Markers and the deletion of specific versioned objects:

  • For delete operations on an object, MinIO replication also creates the delete marker on the target bucket.

  • For delete operations on versions of an object, MinIO replication also deletes those versions on the target bucket.

MinIO does not replicate objects deleted due to lifecycle management expiration rules. MinIO only replicates explicit client-driven delete operations.

MinIO requires explicitly enabling replication of delete operations using the mc replicate add --replicate flag. This procedure includes the required flags for enabling replication of delete operations and delete markers. See Replication of Delete Operations for more complete documentation on this behavior.

Replication of Encrypted Objects

MinIO supports replicating objects encrypted with automatic Server-Side Encryption (SSE-S3). Both the source and destination buckets must have automatic SSE-S3 enabled for MinIO to replicate an encrypted object.

As part of the replication process, MinIO decrypts the object on the source bucket and transmits the unencrypted object. The destination MinIO deployment then re-encrypts the object using the destination bucket SSE-S3 configuration. MinIO strongly recommends enabling TLS on both source and destination deployments to ensure the safety of objects during transmission.

MinIO does not support replicating client-side encrypted objects (SSE-C).

Examples

See the following tutorials for more complete procedures on configuring server-side replication with mc replicate:

Add a New Replication Rule

Use mc replicate add to add a new replication rule to a bucket or bucket prefix. mc replicate depends on the ARN resource returned by mc admin bucket remote.

mc replicate add ALIAS/PATH \
   --arn ARN \
   --remote-bucket BUCKET \
   [--FLAGS]
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the path to the bucket or bucket prefix on which to add the new rule.

  • Replace ARN with the ARN of the remote bucket target created by mc admin bucket remote.

  • Replace BUCKET with the name of the remote bucket target. The specified bucket name must match the ARN bucket.

Include all other optional flags.

Modify an Existing Replication Rule

Use mc replicate edit to modify an existing replication rule.

mc replicate edit ALIAS/PATH \
   --id ID \
   [--FLAGS]
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the path to the bucket or bucket prefix on which the rule exists.

  • Replace ID with the unique identifier for the rule to modify. Use mc replicate ls to retrieve the list of replication rules on the bucket and their corresponding identifiers.

Important

MinIO applies replication rules to objects as part of write operations. Modifying a replication rule has no effect on existing objects in the bucket. For example, enabling delete marker replication through the --replicate option does not automatically replicate existing delete markers or deleted object versions.

Disable or Enable an Existing Replication Rule

Use mc replicate edit with the --state flag to disable or enable a replication rule.

mc replicate edit ALIAS/PATH \
   --id ID \
   --state "disabled"|"enabled"
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the path to the bucket or bucket prefix on which the rule exists.

  • Replace ID with the unique identifier for the rule to modify. Use mc replicate ls to retrieve the list of replication rules on the bucket and their corresponding identifiers.

  • Specify either "disabled" or "enabled" to the state flag to disable or enable the replication rule.

Important

MinIO applies replication rules to objects as part of write operations. Modifying a replication rule has no effect on existing objects in the bucket. In context of enabling or disabling a replication rule, objects written to a bucket with no enabled replication rules are not automatically replicated if one or more rules are enabled later.

Remove a Replication Rule

Use mc replicate rm to remove an existing replication rule:

mc replicate rm ALIAS/PATH --id ID
  • Replace ALIAS with the alias of the MinIO deployment.

  • Replace PATH with the path to the bucket or bucket prefix on which the rule exists.

  • Replace ID with the unique identifier for the rule to modify. Use mc replicate ls to retrieve the list of replication rules on the bucket and their corresponding identifiers.

Important

MinIO applies replication rules to objects as part of write operations. Deleting a replication rule has no effect on objects replicated as part of that rule.

Syntax

mc replicate add

Adds a new server-side replication configuration rule for a bucket. Requires specifying the resource returned by mc admin bucket remote.

mc replicate add has the following syntax:

mc replicate add SOURCE \
   --arn ARN \
   --remote-bucket DESTINATION \
   --replicate OPTIONS \
   [FLAGS]

mc replicate add supports the following arguments:

SOURCE

Required

The full path to the bucket on which to add the bucket replication configuration. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate add play/mybucket
--arn

Deprecated in RELEASE.2021-09-23T05-44-03Z. mc replicate add --remote-bucket supersedes all functionality provided by this option.

--remote-bucket

Required

Specify the ARN for the destination deployment and bucket. You can retrieve the ARN using mc admin bucket remote:

The specified ARN bucket must match the value specified to --remote-bucket.

Added in RELEASE.2021-09-23T05-44-03Z. Requires MinIO server RELEASE.2021-09-23T04-46-24Z.

--replicate

Optional

Specify a comma-separated list of the following values to enable extended replication features.

  • delete - Directs MinIO to replicate DELETE operations to the destination bucket.

  • delete-marker - Directs MinIO to replicate delete markers to the destination bucket.

  • existing-objects - Directs MinIO to replicate objects created before replication was enabled or while replication was suspended.

--tags

Optional

Specify one or more ampersand & separated key-value pair tags which MinIO uses for filtering objects to replicate. For example:

--tags "TAG1=VALUE&TAG2=VALUE&TAG3=VALUE"

MinIO applies the replication rule to any object whose tag set contains the specified replication tags.

--id

Optional

Specify a unique ID for the replication rule. MinIO automatically generates an ID if one is not specified.

--priority

Optional

Specify the integer priority of the replication rule. The value must be unique among all other rules on the source bucket. Higher values imply a higher priority than all other rules.

The default value is 0.

--storage-class

Optional

Specify the MinIO storage class to apply to replicated objects.

--insecure

Optional

Disables verification of the destination deployment’s TLS certificate. This option may be required if the destination deployment uses a self-signed certificate or a certificate signed by an unknown Certificate Authority.

--disable

Optional

Creates the replication rule in the “disabled” state. MinIO does not begin replicating objects using the rule until it is enabled using mc replicate edit.

Objects created while replication is disabled are not immediately eligible for replication after enabling the rule. You must explicitly enable replication of existing objects by including "existing-objects" to the list of replication features specified to mc replicate edit --replicate. See Replication of Existing Objects for more information.

mc replicate edit

Modifies an existing server-side replication configuration rule for a bucket.

mc replicate edit has the following syntax:

mc replicate edit SOURCE --id IDENTIFIER [FLAGS]

mc replicate edit supports the following arguments:

SOURCE

Required

The full path to the bucket on which to edit the bucket replication configuration. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate edit play/mybucket
--id

Required

Specify the unique ID for a configured replication rule.

--remote-bucket

Optional

Specify the name of the bucket on the destination deployment. The name must match the replication rule ARN. Use mc replicate ls to validate the ARN for each configured replication rule on the bucket.

--replicate

Optional

Specify a comma-separated list of the following values to enable extended replication features:

  • delete - Directs MinIO to replicate DELETE operations to the destination bucket.

  • delete-marker - Directs MinIO to replicate delete markers to the destination bucket.

  • replica-metadata-sync - Directs MinIO to synchronize metadata-only changes on a replicated object back to the source. This feature only effects two-way active-active replication configurations.

    Omitting this value directs MinIO to stop replicating metadata-only changes back to the source.

  • existing-objects - Directs MinIO to replicate objects created prior to configuring or enabling replication. MinIO by default does not synchronize existing objects to the remote target.

    See Replication of Existing Objects for more information.

--tags

Optional

Specify one or more ampersand & separated key-value pair tags which MinIO uses for filtering objects to replicate. For example:

--tags "TAG1=VALUE&TAG2=VALUE&TAG3=VALUE"

MinIO applies the replication rule to any object whose tag set contains the specified replication tags.

--priority

Optional

Specify the integer priority of the replication rule. The value must be unique among all other rules on the source bucket. Higher values imply a higher priority than all other rules.

--storage-class

Optional

Specify the MinIO storage class to apply to replicated objects.

--insecure

Optional

Disables verification of the destination deployment’s TLS certificate. This option may be required if the destination deployment uses a self-signed certificate or a certificate signed by an unknown Certificate Authority.

--state

Optional

Enables or disables the replication rule. Specify one of the following values:

  • "enable" - Enables the replication rule.

  • "disable" - Disables the replication rule.

Objects created while replication is disabled are not immediately eligible for replication after enabling the rule. You must explicitly enable replication of existing objects by including "existing-objects" to the list of replication features specified to mc replicate edit --replicate. See Replication of Existing Objects for more information.

mc replicate ls

Lists the server-side replication configuration rules for a bucket.

mc replicate ls has the following syntax:

mc replicate ls SOURCE [FLAGS]

mc replicate ls supports the following arguments:

SOURCE

Required

The full path to the bucket on which to list the replication configurations. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate ls play/mybucket
--insecure

Optional

Disables verification of the destination deployment’s TLS certificate. This option may be required if the destination deployment uses a self-signed certificate or a certificate signed by an unknown Certificate Authority.

--status

Optional

Filter replication rules on the bucket based on their status. Specify one of the following values:

  • enabled - Show only enabled replication rules.

  • disabled - Show only disabled replication rules.

If omitted, mc replicate ls defaults to showing all replication rules.

mc replicate export

Exports all server-side replication configuration rules for a bucket as a JSON document.

mc replicate export has the following syntax:

mc replicate export SOURCE [FLAGS]

mc replicate export supports the following arguments:

SOURCE

Required

The full path to the bucket for which to export the replication configurations. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate export play/mybucket
--insecure

Optional

Disables verification of the destination deployment’s TLS certificate. This option may be required if the destination deployment uses a self-signed certificate or a certificate signed by an unknown Certificate Authority.

mc replicate import

Imports JSON-formatted server-side replication rules for a bucket through STDIN.

mc replicate import has the following syntax:

mc replicate import SOURCE [FLAGS]

mc replicate import also supports input redirection for specifying the path to the JSON-formatted rules:

mc replicate import SOURCE [FLAGS] < /path/to/config

mc replicate import supports the following arguments:

SOURCE

Required

The full path to the bucket to which to import the replication configurations. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate import play/mybucket
--insecure

Optional

Disables verification of the destination deployment’s TLS certificate. This option may be required if the destination deployment uses a self-signed certificate or a certificate signed by an unknown Certificate Authority.

mc replicate rm

Removes one or more server-side replication rules on a bucket.

mc replicate rm has the following syntax:

mc replicate rm SOURCE --id ID [FLAGS]

mc replicate rm supports the following arguments:

SOURCE

Required

The full path to the bucket on which to remove the bucket replication configuration. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate edit play/mybucket
--id

Optional

Specify the unique ID for a configured replication rule.

--all

Removes all replication rules on the specified bucket. Requires specifying the --force flag.

--force

Optional

Required if specifying --all .

mc replicate resync, reset

Resynchronizes all objects in the specified bucket to the remote target bucket. See Resynchronization for more complete documentation.

mc replicate resync has the following syntax:

mc replicate resync SOURCE [args]

mc replicate resync supports the following arguments:

SOURCE

Required

The full path to the bucket on which to resync the bucket replication status. Specify the alias of a configured MinIO service as the prefix to the SOURCE path. For example:

mc replicate resync play/mybucket
older-than

Optional

Specify a duration in days where MinIO only resynchronizes objects older than the specified duration.