Batch Job Expiration

The AIStor Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the AIStor server to take advantage of the server-side processing power without constraints of the local machine where you run the AIStor Client.

The expire batch job applies Automatic Object Expiration behavior to a single bucket. The job determines expiration eligibility based on the provided configuration, independent of any configured expiration rules.

Batch expiration supports the following types of rules to find objects to expire:

• object applies only to objects that do not have a delete marker.
• deleted applies only to objects that do have a delete marker.
• prefix applies to all objects with a specified prefix.

Use up to fifty (50) rules of type object or deleted to expire objects that match a set of filters. Batch jobs apply rules individually.

The batch job checks each object against each of the rules in the listed order. Within each rule, an object must satisfy all of the rule’s filters to match. Once an object matches a rule, the job expires the object with no further rule evaluation applied. If no rules match an object, the job retains the object.

The retainVersions filter works fastest when set to 0 to delete all versions, as it does not examine each individual object version.

Use a single rule of type: prefix to expire all objects in the job’s listed prefix(es). This type runs faster than the other job expiration types because it does not enumerate individual objects.

The prefix rule type does not allow for any filters and must be the only rule defined for a job. You must specify purge.everything in the job template to confirm that you want to expire all objects in the listed prefix(es).

Batch expiration occurs immediately as part of the batch job, as compared to the passive scanner-based application of expiration rules. Specifically, batch expiration does not yield to application I/O and may impact performance of regular read/write operations on the deployment.

The batch expiration works per-bucket and runs once to completion. The job determines expiration eligibility at the time the job runs, and does not re-scan or re-check for new objects periodically.

To capture any new objects eligible for expiration, re-run the batch job.

The batch expiration job checks only the latest or “current” version of each object against each batch expiration rule.

As of AIStor RELEASE.2025-08-13T17-08-54Z, you can define batch expiration jobs to look in multiple prefixes in the bucket for matching objects. AIStor expires all objects that match the criteria of any of the job’s rules in any of the listed prefixes.

To specify a single prefix for the job’s rule(s) to apply to, add the prefix in the YAML similar to:

prefix: myprefix

To specify multiple prefixes for the job’s rule(s) to apply to, use a YAML list similar to:

prefix:
  - myprefix1
  - myprefix2
  ...

The job applies to each prefix in the order specified. There is no limit to the number of prefixes you can apply a batch expiration job to.

Batch expiration cannot run on a bucket with object locking enabled. This is the case even if no objects have an active retention period or legal hold applied.

Use mc batch generate to create a basic expire batch job for further customization.

expire:
  apiVersion: v1
  bucket: mybucket # Bucket where this job will expire matching objects from
  prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
  rules:
    - type: object  # objects with zero or more older versions
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 70h # match objects older than this value
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      tags:
        - key: name
          value: pick* # match objects with tag 'name', all values starting with 'pick'
      metadata:
        - key: content-type
          value: image/* # match objects with 'content-type', all values starting with 'image/'
      size:
        lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
        greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object.

    - type: deleted # objects with delete marker as their latest version
      name: NAME # match object names that satisfy the wildcard expression.
      olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
      createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
      purge:
          # retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
          # retainVersions: 5 # keep the latest 5 versions of the object including delete markers.

  notify:
    endpoint: https://notify.endpoint # notification endpoint to receive job completion status
    token: Bearer xxxxx # optional authentication token for the notification endpoint

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: 500ms # least amount of delay between each retry

The following example expires and purges all objects and versions marked with the specified prefix list.

expire:
  apiVersion: v1
  bucket: test-bucket-versioned
  # required for this rule type
  prefix:
    - test-dir/0
    - test-dir/1
    - test-dir/3
    - test-dir/4
  rules:
    # cannot specify any other rule type with this one
    - type: prefix
      purge:
        everything: true

When using the type: prefix, the following must be true:

• You cannot combine a type: prefix with any other rule type in the same job. It must be the only rule.
• The job must specify at least one prefix:.
• The rule must also set rules.[n].purge.everything: true.