Batch Job Expiration
The AIStor Batch Framework allows you to create, manage, monitor, and execute jobs using a YAML-formatted job definition file (a “batch file”). The batch jobs run directly on the AIStor server to take advantage of the server-side processing power without constraints of the local machine where you run the AIStor Client.
The expire batch job applies Automatic Object Expiration behavior to a single bucket.
The job determines expiration eligibility based on the provided configuration, independent of any configured expiration rules.
Behavior
Immediate expiration of objects
Batch expiration occurs immediately as part of the batch job, as compared to the passive scanner-based application of expiration rules. Specifically, batch expiration does not yield to application I/O and may impact performance of regular read/write operations on the deployment.
Expiration eligibility Determined at batch-run
The batch expiration works per-bucket and runs once to completion. The job determines expiration eligibility at the time the job runs, and does not re-scan or re-check for new objects periodically.
To capture any new objects eligible for expiration, re-run the batch job.
Expiry rules check latest object only
The batch expiration job only checks the latest or “current” version of each object against each batch expiration rule.
Expire batch job reference
| Field | Description |
|---|---|
expire |
Required Top-level field for the expiration job type. |
apiVersion |
Required Set to v1. |
bucket |
Required Specify the name of the bucket in which the job runs. |
prefix |
Optional Specify the bucket prefix in which the job runs. |
rules |
Required An array of one or more expiration rules to apply to objects in the specified bucket and prefix (if any). |
rules.[n].type |
Required Supports one of the following two values: - object - Applies only to objects which do not have a DeleteMarker as the current version.- deleted - Applies only to objects which do Have a DeleteMarker as the current version.See Object Deletion for more complete documentation on DeleteMarker or delete operations in versioned buckets. |
rules.[n].name |
Optional Specify a match string to use for filtering objects. Supports glob-style wildcards ( *, ?). |
rules.[n].olderThan |
Optional Specify the age of objects for filtering objects. The rule applies to only those objects older than the specified unit of time. For example, 72h or 3d selects objects older than three days. |
rules.[n].createdBefore |
Optional Specify an ISO-8601 timestamp for filtering objects. The rule applies to only those objects created before the specified timestamp. |
rules.[n].tags |
Optional Specify an array of key-value pairs describing object tags to use for filtering objects. The value entry supports glob-style wildcards (*, ?).For example, the following filters the rule to only objects with matching tags: yaml<br/>tags:<br/> - key: archive<br/> value: True<br/>This key is incompatible with rules.[n].type: deleted. |
rules.[n].metadata |
Optional Specify an array of key-value pairs describing object metadata to use for filtering objects. The value key supports glob-style wildcards (*, ?).For example, the following filters the rule to only objects with matching metadata: yaml<br/>metadata:<br/> - key: content-type<br/> value: image/*<br/>This key is incompatible with rules.[n].type: deleted. |
rules.[n].size |
Optional Specify the range of object sizes for filtering objects. - lessThan - matches objects with size less than the specified amount (e.g. MiB, GiB).- greaterThan - matches objects with size greater than the specified amount (e.g. MiB, GiB). |
rules.[n].purge.retainVersions |
Optional Specify the number of object versions to retain when applying expiration. Defaults to 0 for deleting all object versions (fastest). |
notify.endpoint |
Optional The predefined endpoint to send events for notifications. |
notify.token |
Optional An optional JSON Web Token (JWT) to access the notify.endpoint. |
retry.attempts |
Optional The number of tries to complete the batch job before giving up. |
retry.delay |
Optional The amount of time to wait between each attempt ( ms). |
Sample YAML description for an expire job type
Use mc batch generate to create a basic expire batch job for further customization.
expire:
apiVersion: v1
bucket: mybucket # Bucket where this job will expire matching objects from
prefix: myprefix # (Optional) Prefix under which this job will expire objects matching the rules below.
rules:
- type: object # objects with zero ore more older versions
name: NAME # match object names that satisfy the wildcard expression.
olderThan: 70h # match objects older than this value
createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
tags:
- key: name
value: pick* # match objects with tag 'name', all values starting with 'pick'
metadata:
- key: content-type
value: image/* # match objects with 'content-type', all values starting with 'image/'
size:
lessThan: 10MiB # match objects with size less than this value (e.g. 10MiB)
greaterThan: 1MiB # match objects with size greater than this value (e.g. 1MiB)
purge:
# retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
# retainVersions: 5 # keep the latest 5 versions of the object.
- type: deleted # objects with delete marker as their latest version
name: NAME # match object names that satisfy the wildcard expression.
olderThan: 10h # match objects older than this value (e.g. 7d10h31s)
createdBefore: "2006-01-02T15:04:05.00Z" # match objects created before "date"
purge:
# retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.
# retainVersions: 5 # keep the latest 5 versions of the object including delete markers.
notify:
endpoint: https://notify.endpoint # notification endpoint to receive job completion status
token: Bearer xxxxx # optional authentication token for the notification endpoint
retry:
attempts: 10 # number of retries for the job before giving up
delay: 500ms # least amount of delay between each retry