mc table migrate
The mc table migrate command migrates Iceberg table catalog metadata from an external source into AIStor Tables.
Syntax
The command has the following syntax:
mc table migrate [FLAGS] ALIAS WAREHOUSE
Flags
| Flag | Description |
|---|---|
--source-type value |
Source catalog type: rest, hive, glue, sql, nessie. |
--source-uri value |
Source catalog URI (e.g., http://nessie:19120/iceberg). |
--source-warehouse value |
Source warehouse location/name. |
--source-access-key value |
Access key (client ID) for source catalog authentication [$MC_TABLE_MIGRATE_SOURCE_ACCESS_KEY]. |
--source-secret-key value |
Secret key (client secret) for source catalog authentication [$MC_TABLE_MIGRATE_SOURCE_SECRET_KEY]. |
--source-token value |
OAuth token for source catalog [$MC_TABLE_MIGRATE_SOURCE_TOKEN]. |
--source-property value |
Additional source catalog property (key=value), can be repeated. |
--namespaces value |
Comma-separated list of namespaces to migrate (default: all); mutually exclusive with --identifiers, --identifiers-from-file, and --identifiers-regex. |
--tables value |
Regex pattern matched against the table name; mutually exclusive with --identifiers, --identifiers-from-file, and --identifiers-regex (default: “.*”). |
--identifiers value |
Comma-separated list of exact identifiers to migrate (format: namespace.table or namespace.table [view]); mutually exclusive with --namespaces, --tables, --identifiers-from-file, and --identifiers-regex. |
--identifiers-from-file value |
Path to a file with one identifier per line (namespace.table or namespace.table [view]); pass failed_identifiers.txt from a previous run to retry only what failed; mutually exclusive with --identifiers, --namespaces, --tables, and --identifiers-regex. |
--identifiers-regex value |
Regex matched against the full ’namespace.table’ identifier across all namespaces; mutually exclusive with --identifiers, --identifiers-from-file, --namespaces, and --tables. |
--exclude-views |
Skip migration of views. |
--dry-run |
Show what would be done without executing. |
--ignore-existing |
Skip tables/views/namespaces that already exist. |
--delete-source |
Drop each table/view entry from the source catalog after successful registration in the target (does not delete data files). |
--force |
Skip the confirmation prompt required by --delete-source. |
--output-dir value |
Write per-category identifier lists to this directory after migration (only files with entries are written). |
Global flags
This command supports any of the global flags.
Examples
Migrate all namespaces from Nessie to MinIO.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse myminio my-warehouse
Migrate specific namespaces from Nessie, skipping existing tables.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --namespaces analytics,sales --ignore-existing myminio my-warehouse
Dry run to preview what would be migrated.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --namespaces analytics --dry-run myminio my-warehouse
Migrate from a generic REST catalog with credentials supplied via environment variables.
mc table migrate --source-type rest --source-uri http://polaris:8181 --source-warehouse lakehouse myminio my-warehouse
Migrate from a specific Nessie branch, only tables whose names start with “customer”.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --source-property ref=dev --exclude-views --tables "customer.*" myminio my-warehouse
Migrate from Hive Metastore (tables only, Iceberg views not supported in Hive).
mc table migrate --source-type hive --source-uri thrift://hive-metastore:9083 --source-warehouse s3://bucket/warehouse --namespaces analytics myminio my-warehouse
Migrate from AWS Glue Data Catalog.
mc table migrate --source-type glue --source-property glue.region=us-east-1 myminio my-warehouse
Glue takes its explicit configuration through --source-property and does not use --source-uri.
See Glue credentials before supplying access keys this way.
One-way move from Nessie: register in MinIO then drop from Nessie (prompts for confirmation).
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --delete-source myminio my-warehouse
Same as above but non-interactive (e.g. CI pipeline); skip the confirmation prompt with --force.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --delete-source --force myminio my-warehouse
Dry run with output directory to review what would be migrated before committing.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --dry-run --output-dir /tmp/migration-preview myminio my-warehouse
Migrate only specific tables by exact identifier, bypassing namespace discovery.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --identifiers analytics.customers,analytics.orders myminio my-warehouse
Retry a partial run by feeding the failed_identifiers.txt from a previous --output-dir run.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --identifiers-from-file /tmp/migration-preview/failed_identifiers.txt myminio my-warehouse
Migrate all identifiers (tables and views) whose full namespace.table name matches a regex.
mc table migrate --source-type nessie --source-uri http://nessie:19120/iceberg --source-warehouse warehouse --identifiers-regex "analytics\..*" myminio my-warehouse
Glue credentials
A --source-property value becomes an argument in the mc process, where any user who can list processes on the host can read it.
Passing a variable such as $AWS_SECRET_ACCESS_KEY does not avoid this: the shell expands it before mc starts.
Glue takes its explicit configuration through --source-property, so --source-access-key, --source-secret-key, and their MC_TABLE_MIGRATE_SOURCE_ACCESS_KEY and MC_TABLE_MIGRATE_SOURCE_SECRET_KEY environment variables do not apply to it.
Those flags cover the source types that take a URI: rest, hive, sql, and nessie.
Run a Glue migration from a host that already holds AWS credentials the Glue client can find, such as an instance role or a named profile, and pass only glue.region.
Where that is not possible and you must pass keys on the command line, use short-lived credentials and revoke them once the migration finishes.
Behavior
S3 compatibility
The mc command-line tool is built for compatibility with the AWS S3 API and is tested with MinIO AIStor and AWS S3 for expected functionality and behavior.