Object Versioning

MinIO AIStor supports keeping multiple “versions” of an object in a single bucket.

When enabled, versioning allows MinIO AIStor to keep multiple iterations of the same object. Write operations which would normally overwrite an existing object instead result in the creation of a new version of the object. MinIO AIStor versioning protects from unintended overwrites and deletions while providing support for “undoing” a write operation. Bucket versioning is a prerequisite for configuring object locking and retention rules.

For versioned buckets, a write operation that mutates an object results in a new version of that object with a unique version ID. MinIO AIStor marks the “latest” version of the object that clients retrieve by default. Clients can then explicitly choose to list, retrieve, or remove a specific object version by including the desired version ID.

Define object expiration rules to remove versions of objects no longer needed, such as by the number of versions or the date of versions.

Review each of the four images in this series to see how MinIO AIStor retrieves objects in a versioned bucket.

MinIO AIStor adds a unique version ID to each object as part of write operations.

MinIO AIStor retains all versions of an object and marks the most recent version as the “latest”.

A read operation request without a version ID returns the latest version of the object.

Include the version ID to retrieve a specific version of an object during a read operation.

MinIO AIStor does not create versions for creation, mutation, or deletion of explicit directory objects (“prefixes”). Objects created within that explicit directory object retain normal versioning behavior.

MinIO AIStor implicitly determines prefixes from object paths. Explicit prefix creation typically only occurs with Spark and similar workloads which apply legacy POSIX/HDFS directory creation behavior within the S3 context.

MinIO AIStor uses the full namespace (the bucket and path to an object) for each object as part of determining object uniqueness. For example, all of the following namespaces are “unique” objects, where mutations of each object result in the creation of new object versions at that namespace:

databucket/object.blob
databucket/blobs/object.blob
blobbucket/object.blob
blobbucket/blobs/object.blob

While object.blob might be the same binary across all bucket namespaces, MinIO AIStor only enforces versioning within a specific bucket namespace and therefore considers each object.blob above as distinct and unique.

MinIO AIStor does not perform incremental or differential-type versioning. For mutation-heavy workloads, this may result in substantial drive usage by older or aged object versions.

For example, consider a 1GB object containing log data. An application appends 100MB of data to the log and uploads to MinIO AIStor. MinIO AIStor would then contain both the 1GB and 1.1GB versions of the object. If the application repeated this process every day for 10 days, the bucket would eventually contain more than 14GB of data associated to a single object.

MinIO AIStor supports configuring object lifecycle management rules to automatically expire or transition aged object versions and free up storage capacity. For example, you can configure a rule to automatically expire object versions 90 days after they become non-current (that is, no longer the “latest” version of that object). See MinIO AIStor Object Expiration for more information.

You can alternatively perform manual removal of object versions using the following commands:

• mc rm --versions - Removes all versions of an object.
• mc rm --versions --older-than - Removes all versions of an object older than the specified calendar date.

MinIO AIStor alerts on objects that accumulate an unusual number or volume of versions:

• A version count that reaches the alert_excess_versions threshold, which defaults to 100.
• A cumulative size across all versions of a single object that reaches 1 TiB. This threshold is fixed.

Both are notifications only. MinIO AIStor does not block writes, remove versions, or otherwise change behavior when an object crosses either threshold. The alerts exist so that an administrator can find the workload responsible and decide what to do about it.

MinIO AIStor generates a unique and immutable identifier for each versioned object as part of write operations. Each object version ID consists of a 128-bit fixed-size UUIDv4. UUIDv4 values are sufficiently random to make collisions highly unlikely in any environment, and do not require a centralized registration process or authority to guarantee uniqueness.

MinIO AIStor does not support client-managed version ID allocation. All version ID generation is handled by the Object Store process.

For objects created while versioning is disabled or suspended, MinIO AIStor uses a null version ID. You can access or remove these objects by specifying null as the version ID as part of S3 operations.

Performing a DELETE operation on a versioned object creates a 0-byte DeleteMarker as the latest version of that object. For objects where the latest version is a DeleteMarker, clients must specify versioning flags or identifiers to perform GET/HEAD/LIST/DELETE operations on a prior version of that object. The default server behavior omits DeleteMarker objects from consideration for unversioned operations.

MinIO AIStor can utilize Lifecycle Management expiration rules to automatically remove versioned objects permanently. Otherwise, use manual DELETE operations to permanently remove non-current versioned objects or DeleteMarker objects.

To permanently delete an object version, perform the DELETE operation and specify the version ID of the object to delete. Versioned delete operations are irreversible.

Performing a DELETE operation on a versioned object produces a DeleteMarker for that object.

Clients by default retrieve the “latest” object version. MinIO AIStor returns a 404-like response if the latest version is a DeleteMarker.

Clients can retrieve any previous version of the object by specifying the version ID, even if the “Latest” version is a DeleteMarker.

Clients can delete a specific object version by specifying the version ID as part of the DELETE operation. Deleting a specific version is permanent and does not result in the creation of a DeleteMarker.

The following mc commands operate on DeleteMarkers or versioned objects:

• Use mc ls --versions to view all versions of an object, including delete markers.
• Use mc cp --version-id=UUID ... to retrieve the version of the “deleted” object with matching UUID.
• Use mc rm --version-id=UUID ... to delete the version of the object with matching UUID.
• Use mc rm --versions to delete all versions of an object.