Site Replication and Disaster Recovery

AIStor Tables integrates with site replication so that your Iceberg catalog and warehouse data are available on a disaster-recovery (DR) site.

When site replication is configured, AIStor Tables replicates the following to every peer site:

• Warehouses and their catalogs
• Namespaces
• Tables and their data
• Default warehouse encryption configuration

The replica site runs a catalog scanner that rebuilds its Iceberg catalog from the replicated data. The scanner is off by default; you turn it on to make a site the replica. Once it is on, the replica converges on the primary without further intervention. See Set up the replica site.

Staged tables do not replicate, because they are not committed. Commit a table on the primary to make it eligible.

Make the DR site a replica before you configure site replication. A site that is not a replica rejects inbound replication to warehouse buckets, so tables never arrive.

Before you begin:

• Only one site may hold data at setup. This is a site replication requirement covering all buckets and objects, not warehouses alone, and the command must be run against the site that holds it — so the site holding your warehouses is always the primary.
• If the primary already holds warehouses, check whether they need a catalog identity backfill and run it first.
• Confirm the DR site holds no warehouses of its own. Site replication refuses to add a peer that already has one, because tables data flows only from primary to replica.

1. Make the DR site the replica:

   mc admin config set myreplica tables replica_catalog=on
   

   The change takes effect immediately, with no restart.

   MinIO AIStor refuses this if another site in the configuration is already the replica, or if the configuration holds more than two sites.

2. Configure site replication, naming the primary first:

   mc admin replicate add myprimary myreplica
   

   mc sends the request to the first alias, and site replication rejects any other site in the list that already holds buckets.

   For the full site replication procedure and its prerequisites, see Initialize site replication.

3. Confirm the roles on both sites:

   mc admin config get myprimary tables | grep replica_catalog=
   mc admin config get myreplica tables | grep replica_catalog=
   

   The primary must report replica_catalog=off and the replica replica_catalog=on.

The replica’s catalog fills in as warehouse data arrives. Whether a table the scanner has accepted is queryable on the replica depends on the consistency mode: strict accepts a version only once every file is present, while eventual can accept one whose data files are still arriving. Track that with mc table replicate status.

The scanner’s defaults suit most deployments. The one setting worth a decision is replica_catalog_consistency, which decides whether the replica can serve queries on its own or depends on the primary to serve files that have not arrived yet.

For that setting and the rest of the scanner’s tuning values, see Replica catalog settings.

Warehouses created before AIStor supported Tables replication do not carry the per-table catalog identity that the replica’s scanner reads, so their tables cannot be reconstructed on the replica. A backfill writes that identity. Warehouses created since then already carry it.

Run the backfill against the primary, before you configure site replication:

mc table replicate backfill start myprimary
mc table replicate backfill status myprimary

Cancel a running backfill with mc table replicate backfill cancel myprimary.

Backfill writes metadata only. It does not modify table data files or metadata.json bodies, and it preserves existing object versions and ETags. It is idempotent, so re-running it is safe, and only one backfill runs at a time per cluster.

If site replication is already configured but no site was ever made the replica, warehouse data never replicated. Setting the role does not fetch what was missed, so designate the replica and then resync:

mc admin config set myreplica tables replica_catalog=on
mc admin replicate resync start myprimary myreplica
mc admin replicate resync status myprimary myreplica

Warehouses that already exist keep working; only the data written while the role was unset is missing on the peer.

A replica site accepts inbound replication but rejects client writes.

Run mc table replicate status against the replica site. Each site reports only its own replication state, and only the replica runs the catalog scanner:

mc table replicate status myreplica

The output reports the scanner state, then every table and view grouped by warehouse and namespace, with the version the replica has verified, the latest version it has seen, and any files it is waiting on.

On a healthy replica:

• Scanner reads active or tracking, depending on the consistency mode.
• Every table reads Status: OK.

A table reading N versions behind is normal while a new version’s files are still replicating. Treat it as a problem when the count does not fall across several scan cycles, or when Missing Files names the same file each time. In that case, resynchronize the object data from the primary with mc admin replicate resync, then watch the count again.

For the full field reference, including every Scanner value, see mc table replicate status.

To alert on a stalled or failing scanner instead of polling it, see Replica catalog scanner metrics.

If the primary site becomes unavailable, promote the DR site to accept write traffic.

Promote the DR site:

mc table replicate failover myreplica

The command signals the replica’s catalog scanner to run a final scan that applies any replicated state still pending, then returns. Promotion completes asynchronously, when that scan finishes. The scanner then disables itself, and the promoted site begins accepting writes to warehouse buckets.

Monitor progress with mc table replicate status myreplica.

After promotion, the new primary rejects any further replication to warehouse buckets, so an old primary that comes back cannot push stale writes into it.

No single recovery path fits every deployment. Choose between two.

Either decommission the old primary, or demote it to be the new replica with the catalog resync below.

Before you start, assess any data that diverged during the failover window and copy anything you need to keep to the new primary.

Complete option 1 first, so both sites hold identical converged data. Then swap the roles during scheduled downtime, with client writes stopped on the promoted site:

mc admin config set myprimary tables replica_catalog=off
mc admin config set myreplica tables replica_catalog=on

Run them in that order. Setting replica_catalog=on is refused while another site is already the replica, so demoting the promoted site first would fail — promote the old primary first, then demote.

Between the two commands both sites are primaries, and two primaries accept divergent table writes. That is why client writes must be stopped for the whole window. Wait for the first command to take effect, confirm with mc admin config get myprimary tables, then run the second and check both roles.

Then resume client writes, directed at the old primary.

The catalog resync rebuilds the catalog on the site you are demoting so that it converges on the new primary, and preserves everything the new primary holds. You drive it in two phases.

In the steps below, myprimary is the old primary you are demoting, and myreplica is the site that failover promoted. Failover swapped the sites’ roles, not their aliases.

1. Open a resync window on the site you are demoting. It then accepts inbound warehouse replication again, while still rejecting client writes. The catalog is left untouched in this phase:

   mc table replicate resync open myprimary
   

   This step is what reopens inbound replication. Failover left this site with its scanner off, and a site without the scanner rejects replication to warehouse buckets, so object data cannot reach it until the window is open.

2. Resync object data from the new primary, and wait for it to finish:

   mc admin replicate resync start myreplica myprimary
   mc admin replicate resync status myreplica myprimary
   

3. Rebuild the catalog:

   mc table replicate resync rebuild myprimary
   

   The command refuses to start while the new primary still reports a resync in progress, so confirm step 2 has finished. It prompts for confirmation; pass --yes to skip the prompt in automation.

   The rebuild backs up and deletes the local catalog, then runs one scan cycle that checks every table and view against the new primary:

   • Warehouse and namespace properties are taken from the new primary, so namespace membership and warehouse encryption defaults come from the authoritative site.
   • Table metadata versions that exist only on the demoted site are discarded, walking back to the newest version the new primary also holds.
   • A table or view that only ever existed on the demoted site is marked deleted. Its data files stay on disk.
   • A table deleted only on the demoted site, but still live on the new primary, is restored.

   On success the site enables its catalog scanner and becomes a full replica.

Monitor the rebuild with mc table replicate status myprimary. While the window is open, that site reports Scanner: rebuilding and a Rebuild state.

If the primary is gone for an extended period, provision a new cluster and bootstrap it as the replica: remove the site replication configuration and set up Tables replication from scratch.

If a replica’s catalog becomes corrupt or drifts out of sync, reset it and let the scanner rebuild it from scratch:

mc table replicate catalog reset myreplica

The command backs up the live catalog to a timestamped prefix, deletes it, and rebuilds on the next scanner cycle. Restoring the backup requires manual intervention; there is no automated restore. The command prompts for confirmation; pass --yes to skip the prompt.

Use this only when guided by the recovery workflow above or by MinIO support.

The catalog scanner and the backfill process emit trace events under the tables-scan trace type. Run mc admin trace against the replica for a live stream of scanner activity:

mc admin trace -v --call tables-scan myreplica

Use -v. Without it, the output shows the event name, path, and duration, but omits the msg field that explains what the scanner decided.

Most scanner events carry a function name that begins with catalogScanner., a path naming the warehouse or table involved, and a msg field. Backfill also emits backfillAllWarehouses, backfillTableIdentity, and backfillViewIdentity under the same trace type.

The scanner itself runs only on the cluster leader, so its events all come from one node. Other work reported under tables-scan can come from any node.

Filter to one kind of event with --funcname, which accepts a glob:

mc admin trace -v --call tables-scan --funcname 'catalogScanner.cycle' myreplica

The events most useful when diagnosing replication:

Iceberg REST API calls such as LoadTable and CreateTable use a separate trace type:

mc admin trace -v --call tables myreplica

If object replication works but no tables reach the replica, check the role first — a site that is not a replica has its warehouse writes rejected by the replication guard, and that is the most common cause.

If the role is correct, the other causes are a scanner that is not reporting yet (see The scanner is not running) and warehouses that predate Tables replication and were never backfilled (see Prepare existing warehouses).

Check the role:

mc admin config get myreplica tables | grep replica_catalog=

replica_catalog=off on the site you expect to be the replica confirms it. This is what happens when site replication is configured before the DR site is made a replica: site replication itself reports success, and only the warehouse buckets fail.

To see the rejections themselves, trace them. The guard names them s3.ReplicationRequest and answers with HTTP 403:

mc admin trace -v --funcname 's3.ReplicationRequest' --status-code 403 myreplica

Set the role, then resync to deliver the warehouse data written while it was unset:

mc admin config set myreplica tables replica_catalog=on
mc admin replicate resync start myprimary myreplica

Setting the role alone does not fetch what was missed.

Read Missing Files in mc table replicate status. When the same file is named cycle after cycle, replication of that object never completed.

The scanner asks the primary to retry a file once it has been missing for longer than replica_catalog_retry_interval. Watch those requests:

mc admin trace -v --call tables-scan --funcname 'catalogScanner.repl-retry' myreplica

A retry cannot help a file the primary believes it already replicated. If the same file is still missing after several retry intervals, resynchronize it instead:

mc admin replicate resync start myprimary myreplica

A table reading Verified: -1 is a different case. It is held back until every member of its multi-table transaction arrives. catalogScanner.txn events report the transaction and, once it exceeds replica_catalog_txn_stale_timeout, that the surviving members were let through.

mc table replicate status reads disabled on a site whose scanner is off, and waiting on one that has not begun reporting.

Confirm the setting:

mc admin config get myreplica tables

If replica_catalog reads off on the site you expect to be the replica, the site is a primary. After a failover this is correct: promotion turns the scanner off. See Recover after a failover.

waiting after a leader change is expected, because the scanner state is held in memory and starts empty on the new leader. If it persists, check the leader’s logs and the scanner metrics. See Replica catalog scanner metrics.