Batch Job Expiration

The AIStor Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the AIStor server to take advantage of the server-side processing power without constraints of the local machine where you run the AIStor Client.

The expire batch job applies Automatic Object Expiration behavior to a single bucket. The job determines expiration eligibility based on the provided configuration, independent of any configured expiration rules.

Behavior

Immediate expiration of objects

Batch expiration occurs immediately as part of the batch job, as compared to the passive scanner-based application of expiration rules. Specifically, batch expiration does not yield to application I/O and may impact performance of regular read/write operations on the deployment.

Expiration eligibility determined at batch-run

The batch expiration works per-bucket and runs once to completion. The job determines expiration eligibility at the time the job runs, and does not re-scan or re-check for new objects periodically.

To capture any new objects eligible for expiration, re-run the batch job.

Expiration rules check latest object only

The batch expiration job checks only the latest or “current” version of each object against each batch expiration rule.

Prefix expiration rule expires all objects with prefix

In AIStor RELEASE.2025-08-13T17-08-54Z and later versions, you can write an expiration rule to expire all objects marked with one or more bucket prefixes. You can further refine the rule to filter out object versions or completely purge all objects with the specified prefixes.

Expire batch job reference

Field	Description
`expire`	Required Top-level field for the expiration job type.
`apiVersion`	Required Set to `v1`.
`bucket`	Required Specify the name of the bucket in which the job runs.
`prefix`	Optional Specify the bucket prefix in which the job runs. Required if `rules.[n].type` = `prefix`
`rules`	Required An array of one or more expiration rules to apply to objects in the specified `bucket` and `prefix` (if any).
`rules.[n].type`	Required Supports one of the following values: - `prefix` - Expire all objects with the specified `prefix`. Cannot be combined with other rule types. - `object` - Applies only to objects that do not have a `DeleteMarker` as the current version. - `deleted` - Applies only to objects that do have a `DeleteMarker` as the current version. See Object Deletion for more complete documentation on `DeleteMarker` or delete operations in versioned buckets.
`rules.[n].name`	Optional Specify a match string to use for filtering objects. Supports glob-style wildcards (`*`, `?`).
`rules.[n].olderThan`	Optional Specify the age of objects for filtering objects. The rule applies to only those objects older than the specified unit of time. For example, `72h` or `3d` selects objects older than three days.
`rules.[n].createdBefore`	Optional Specify an ISO-8601 timestamp for filtering objects. The rule applies to only those objects created before the specified timestamp.
`rules.[n].tags`	Optional Specify an array of key-value pairs describing object tags to use for filtering objects. The `value` entry supports glob-style wildcards (`*`, `?`). For example, the following filters the rule to only objects with matching tags: `yaml<br/>tags:<br/> - key: archive<br/> value: True<br/>` This key is incompatible with `rules.[n].type: deleted`.
`rules.[n].metadata`	Optional Specify an array of key-value pairs describing object metadata to use for filtering objects. The `value` key supports glob-style wildcards (``, `?`). For example, the following filters the rule to only objects with matching metadata: `yaml<br/>metadata:<br/> - key: content-type<br/> value: image/<br/>` This key is incompatible with `rules.[n].type: deleted`.
`rules.[n].size`	Optional Specify the range of object sizes for filtering objects. - `lessThan` - matches objects with size less than the specified amount (e.g. `MiB`, `GiB`). - `greaterThan` - matches objects with size greater than the specified amount (e.g. `MiB`, `GiB`).
`rules.[n].purge.retainVersions`	Optional Specify the number of object versions to retain when applying expiration. Defaults to `0` for deleting all object versions (fastest). Mutually exclusive with `rules.[n].purge.Everything`
`rules.[n].purge.Everything`	Required if `rules.[n].type` = `prefix` Helper rule for expiring all objects with the specified prefix. Mutually exclusive with `rules.[n].purge.retainVersions`.
`notify.endpoint`	Optional The predefined endpoint to send events for notifications.
`notify.token`	Optional An optional JSON Web Token (JWT) to access the `notify.endpoint`.
`retry.attempts`	Optional The number of tries to complete the batch job before giving up.
`retry.delay`	Optional The amount of time to wait between each attempt (`ms`).

Sample YAML description for an `expire` job type

Use mc batch generate to create a basic expire batch job for further customization.

expire:
  apiVersion: v1
  bucket: mybucket # Bucket where this job will expire matching objects from
  prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
  rules:
    - type: object  # objects with zero or more older versions
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 70h # match objects older than this value
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      tags:
        - key: name
          value: pick* # match objects with tag 'name', all values starting with 'pick'
      metadata:
        - key: content-type
          value: image/* # match objects with 'content-type', all values starting with 'image/'
      size:
        lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
        greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object.

    - type: deleted # objects with delete marker as their latest version
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object including delete markers.

  notify:
    endpoint: https://notify.endpoint # notification endpoint to receive job completion status
    token: Bearer xxxxx # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: 500ms # least amount of delay between each retry

Sample YAML description for a `prefix` expiration job type

The following example expires and purges all objects and versions marked with the specified prefix list.

expire:
  apiVersion: v1
  bucket: test-bucket-versioned
  # required for this rule type
  prefix:
    - test-dir/0
    - test-dir/1
    - test-dir/3
    - test-dir/4
  rules:
    # cannot specify any other rule type with this one
    - type: prefix
      purge:
        everything: true