Replica Catalog Settings

This page covers the settings that control the replica catalog scanner, which rebuilds the AIStor Tables Iceberg catalog on a disaster-recovery site. For the setup, monitoring, failover, and recovery procedures that use these settings, see AIStor Tables site replication and disaster recovery.

All settings on this page belong to the tables configuration subsystem. Set them on the replica site only.

You can establish or modify settings by defining:

• an environment variable on the host system prior to starting or restarting the AIStor Server. Refer to your operating system’s documentation for how to define an environment variable.
• a configuration setting using mc admin config set.

If you define both an environment variable and the similar configuration setting, MinIO AIStor uses the environment variable value.

replica_catalog is the exception: it has no environment variable. See Replica catalog for why.

Run this against the DR site:

mc admin config set myreplica tables replica_catalog=on

Configuration changes take effect immediately. The AIStor Server does not require a restart.

A site that was not a replica did not receive warehouse data. Setting the role does not fetch what was missed — resync after setting it.

The rest of the settings on this page tune a scanner that replica_catalog=on has already enabled. Set only the keys you want to change, because mc admin config set replaces the value of every key you name:

mc admin config set myreplica tables replica_catalog_consistency=strict

The following table lists the keys, their defaults, and their environment variable equivalents.

Because replica_catalog has no environment variable, you cannot pin a site’s role through the environment file. Use either method for the tuning values.

Specify whether this site runs the replica catalog scanner. This is the single setting that decides a site’s Tables role.

on makes the site a replica: it runs the scanner, accepts inbound warehouse replication, rejects client writes to tables, and sends no warehouse data of its own. off makes it a primary: it accepts table writes, sends warehouse data to the replica, and rejects inbound warehouse replication.

Set the role on the DR site before you configure site replication.

Defaults to off.

This setting has no environment variable, by design. MinIO AIStor writes it itself: mc table replicate failover writes off when it promotes the replica, and mc table replicate resync rebuild writes on when it demotes a site back to a replica. A static value would pin the role against both. MinIO AIStor ignores any environment value set for it. Peer sites also read each other’s stored configuration to detect which site is the replica, and an environment variable is invisible to that check.

MinIO AIStor validates the topology when you set this to on, and again when you add a peer site. Either refuses if another site is already the replica, or if the configuration holds more than two sites. Setting it back to off is not validated, so take care not to leave two sites without a replica between them.

Specify how thoroughly the scanner checks that an Iceberg table’s files have arrived before it accepts a metadata version into the replica catalog. This choice decides whether queries against the replica are guaranteed to succeed.

An Iceberg table’s files form a tree. A metadata.json file names a snapshot’s manifest list, the manifest list names manifests, and each manifest names the data files that hold the rows.

Valid values:

• eventual - Check only that the manifest list has arrived. This is fast and uses few IOPS, but the catalog can point to a version whose manifests or data files have not replicated yet. A query that reads a file which has not arrived fails, unless site replication serves that read from the primary: its automatic proxy-on-404 forwards a GET or HEAD that misses locally to the peer. That fallback works only while the primary is reachable, so eventual does not make the replica independently queryable.
• strict - Check the whole tree: the manifest list, every manifest, and every data and delete file those manifests name. The scanner accepts a version only when every file is present, so every table in the replica catalog is queryable. This costs far more IOPS, because the scanner issues a request per file in the tree, and a large table’s tree holds many files.

In both modes the scanner caches the files it has confirmed, so later cycles re-check far fewer of them. Neither mode checks anything for a table that has no snapshots yet, or for a view.

Choose strict when the replica must serve queries on its own. Choose eventual when the primary stays available to proxy reads, or when scanner IOPS matter more than guaranteed queryability.

Defaults to eventual.

Specify how long the scanner waits between full scan cycles over all warehouse buckets. A shorter interval reduces how far the replica catalog lags behind the primary, and raises the load the scanner places on the cluster.

The minimum value is 1s.

To pause the scanner without disabling it, set a long interval such as 1h. Disabling the scanner instead makes the site reject inbound replication.

Defaults to 1m.

Specify how long a file must be missing on the replica before the scanner asks the primary site to replicate it again. The retry re-queues the object only when its replication never completed. If the primary already recorded the object as replicated, use mc replicate resync instead.

This setting matters mainly with replica_catalog_consistency=strict, because only strict mode detects missing data files.

The minimum value is 1s.

Defaults to 30m.

Specify how many workers scan one warehouse bucket’s contents in parallel. The scanner splits a bucket’s tables into prefix groups and works on this many groups at a time. More workers finish a cycle faster and consume more cluster resources.

The minimum value is 1.

Defaults to 4.

Specify how long the scanner pauses between tables while it walks their metadata. Raise this value to throttle the I/O the scanner generates on a busy cluster. The scanner skips this pause during failover.

The value must not be negative.

Defaults to 10ms.

Specify how long an incomplete multi-table transaction blocks its member tables before the scanner syncs them one at a time.

A multi-table transaction commits its tables together, so the scanner holds all members back until every one has replicated. If a member was deleted on the primary, it never arrives, and the surviving members would wait forever. After this timeout, the scanner assumes the missing member is gone and lets the rest through. This trades strict atomicity on the replica for progress.

The minimum value is 1s.

Defaults to 6h.