AI Workloads

An AI pipeline reaches the object store in a handful of specific ways. It reads training data, queries structured data as tables, writes checkpoints, and shares the cluster with other jobs doing the same. This page explains which AIStor mechanism serves each of those, how it works, and what it requires.

AIStor stores the data. It does not train models or run inference. The mechanisms below are about getting bytes to and from the accelerators that do.

S3 over RDMA removes one copy from the read path. It helps when a pipeline reads large objects into GPU memory and that copy, not the storage, is the bottleneck. The client registers a buffer and sends its descriptor with the request, and the server transfers the object through that buffer directly. The payload never passes through the kernel network stack.

Three operations have an RDMA path: GetObject, PutObject, and UploadPart. Every other operation uses HTTP, whatever the request headers say.

RDMA requires a deployment running the RDMA build of AIStor Server, an RDMA-capable NIC on the client, and an InfiniBand or RoCE fabric between them. Your application chooses where the object lands. It allocates a buffer and registers it, and the transfer targets that buffer. A GPU is required only when the buffer is in device memory. A transfer into host memory takes the same RDMA path without one. Against a standard deployment, an RDMA request is declined and the client falls back to HTTP.

RDMA runs on two independent paths, and the one above is only the first. S3 over RDMA carries object data between the server and a buffer the client registered, in GPU memory or in host memory. Inter-node RDMA carries erasure-coded shards between AIStor nodes, needs an RDMA fabric but no GPU, and is enabled separately. See RDMA acceleration for both paths and the fabric requirements.

Feature tables, labels, and run metadata are usually tabular, and a pipeline queries them rather than reading them whole. AIStor Tables serves an Apache Iceberg REST catalog from the object store itself. There is no separate catalog service or metadata database to run alongside it.

Query engines connect to that catalog directly. See Connecting query engines to AIStor Tables for the supported engines and their catalog configurations.

Creating a warehouse also creates a bucket to hold the table data and metadata. You manage the contents of that bucket through the Tables API or the mc table commands, not through general S3 operations. That protects the integrity of the objects underneath.

Training, evaluation, and serving jobs on one cluster compete for the same request budget. Bucket-level Quality of Service bounds what any one of them can take.

QoS has two enforcement layers. Inside the server, API QoS caps requests per second and concurrent operations per bucket. In front of the server, network QoS caps bandwidth. It is the only layer that can, because bandwidth has to be shaped before the request body is accepted or the response starts streaming.

Two other controls work alongside it. Bucket quotas bound how much data a bucket holds. The scanner evaluates them periodically rather than in real time, so a bucket can pass its quota between cycles. The multi-tenancy guidance covers isolating teams with IAM policies scoped to buckets or prefixes.

A result is only reproducible if you can still read the exact bytes the run read. On a versioned bucket, every write creates a new version with its own version ID. A read that names a version ID returns that version rather than the latest. Record the version IDs a run consumed. Those inputs stay readable only while the versions themselves are retained, so exclude them from lifecycle expiration, or lock them, when a rerun has to see the same bytes.

When a dataset has to survive deletion attempts as well as overwrites, object locking and immutability enforces Write-Once Read-Many protection on versioned objects. A GOVERNANCE or COMPLIANCE retention mode protects an object for a fixed period. A legal hold protects it indefinitely.

Pipelines that build a manifest before a run need to know what a bucket contains without walking it themselves. Bucket inventory reports produce that listing on a schedule. Each report carries object metadata such as size, last modified date, storage class, encryption status, tags, and user metadata.

Reports are written as CSV, JSON, or Parquet, and can be filtered by prefix, age, size, name pattern, tags, or custom metadata. A pipeline reads the report as a file and treats it as the dataset manifest.

Checkpoints accumulate faster than anything else a training job writes. Object Lifecycle Management expires objects after a set period or after a set number of versions. A checkpoint prefix stays bounded without a cleanup job of your own.

Data preparation often involves tools that read and write files rather than call an S3 API. Those tools reach the same buckets over SFTP, FTPS, or FTP. They authenticate against the same identity providers and policies as any S3 client.

File access has narrower semantics than the S3 API. Clients see only the latest version of an object, and append and rename are not supported.