Batch Framework
The AIStor Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the MinIO deployment to take advantage of the server-side processing power without constraints of the local machine where you run the MinIO Client.
A batch file defines one job task.
Once started, the AIStor server starts processing the batch job. Time to completion depends on the resources available to the deployment.
If any portion of the job fails, AIStor retries the job up to the number of times defined in the job definition.
The AIStor Batch Framework supports the following job types:
| Job Type | Description |
|---|---|
| replicate | Perform a one-time replication procedure from one Object Store location to another Object Store location. |
| keyrotate | Perform a one-time process to cycle the sse-s3 or sse-kms cryptographic keys on objects. |
| expire | Perform a one-time immediate expiration of objects in a bucket. |
AIStor Server batch CLI
- Install the AIStor Client
- Define an
aliasfor the object store
The mc batch commands include
| Command Name | Description |
|---|---|
mc batch generate |
Creates a basic YAML-formatted template file for the specified job type. |
mc batch start |
Launches a batch job from a job batch YAML file. |
mc batch list |
Outputs a list of the batch jobs currently in progress on a deployment. |
mc batch status |
Outputs summaries for active, in-progress jobs or any batch job completed in the previous three (3) days. |
mc batch describe |
Outputs the job definition for a specified job ID. |
Access to mc batch
Each batch job executes using the credentials specified in the batch definition. The success of a given batch job depends on those credentials having the appropriate permissions to perform all requested actions.
Permissions
The user executing the batch job must have the following permissions. Alternatively, restrict users from accessing these functions by blocking or limiting access to these actions.
You can assign any of these actions to users independently or in any combination.
| Permission | Description |
|---|---|
admin:ListBatchJobs |
Grants the user the ability to see batch jobs currently in process. |
admin:DescribeBatchJobs |
Grants the user the ability to see the definition details of batch job currently in process. |
admin:StartBatchJob |
Grants the user the ability to start a batch job. The job may be further restricted by the credentials the job uses to access either the source or target deployments. |
admin:CancelBatchJob |
Allows the user to stop a batch job currently in progress. |
ConsoleAdmin policy includes sufficient access to perform all of these types of batch job actions.
Local Deployment
You run a batch job against a particular deployment by passing an alias to the mc batch command.
The deployment you specify in the command becomes the local deployment within the context of that batch job.