S3 Fan-Out Extension

MinIO AIStor implements an S3 extension that performs fan-out uploads: writing multiple, independently-readable objects from a single source data stream in one request.

Fan-out is an extension to the S3 POST Object (PostUpload) operation. Instead of writing one object per uploaded stream, MinIO AIStor reads the source stream once and creates each object described in the fan-out list. Every created object is a complete, standalone object that you can read afterward using the regular S3 GetObject API.

A common use case is fanning out incoming TSB (Time Shift Buffer) media segments. TSBs facilitate time-shifted playback of television signaling and media content, where a single incoming stream must be written out as many separate objects.

Fan-out uploads are automatically enabled when the x-minio-fanout-list form-field is supplied with the PostUpload (S3 POST Object) request. When this form-field is present, MinIO AIStor treats the uploaded body as the shared source stream and creates one object for each entry in the fan-out list.

Each entry in the fan-out list specifies its own object key and may optionally include per-object settings:

• Custom user metadata
• Object tags
• Object retention (mode and retain-until date)

If a fan-out entry fails, the response reports the failure for that specific object while the other objects in the request can still be created.

To avoid building the x-minio-fanout-list form-field by hand, the MinIO SDKs provide a higher-level wrapper, PutObjectFanOut, that constructs the request for you. This wrapper is available in minio-go and minio-java.

You provide the bucket, the single source stream, and a list of fan-out entries. Each entry carries a mandatory object key and any optional per-object settings:

The response returns one result per requested object, including the object key, ETag, and version ID for successful uploads, or an error for any object that could not be created.

The following examples create multiple objects from a single source stream.

package main

import (
    "bytes"
    "context"
    "log"

    "github.com/minio/minio-go/v7"
    "github.com/minio/minio-go/v7/pkg/credentials"
)

func main() {
    client, err := minio.New("localhost:9000", &minio.Options{
        Creds:  credentials.NewStaticV4("minioadmin", "minioadmin", ""),
        Secure: false,
    })
    if err != nil {
        log.Fatal(err)
    }

    source := bytes.NewReader([]byte("shared source stream contents"))

    fanOutReq := minio.PutObjectFanOutRequest{
        Entries: []minio.PutObjectFanOutEntry{
            {Key: "segment-0"},
            {
                Key:         "segment-1",
                UserTags:    map[string]string{"channel": "news"},
                ContentType: "video/mp2t",
            },
        },
    }

    results, err := client.PutObjectFanOut(context.Background(), "my-bucket", source, fanOutReq)
    if err != nil {
        log.Fatal(err)
    }

    for _, r := range results {
        if r.Error != nil {
            log.Printf("failed to create %s: %v", r.Key, r.Error)
            continue
        }
        log.Printf("created %s (etag %s)", r.Key, r.ETag)
    }
}

Map<String, String> tags = new HashMap<>();
tags.put("channel", "news");

PutObjectFanOutResponse response =
    minioClient.putObjectFanOut(
        PutObjectFanOutArgs.builder()
            .bucket("my-bucket")
            .stream(stream, size, -1)
            .entries(
                new PutObjectFanOutEntry[] {
                  PutObjectFanOutEntry.builder().key("segment-0").build(),
                  PutObjectFanOutEntry.builder().key("segment-1").tags(tags).build()
                })
            .build());

After the upload completes, read each created object independently with the standard GetObject API.