Reverse proxies, load balancers, and scaling

Production Delta Sharing deployments place MinIO AIStor behind a reverse proxy, ingress controller, or load balancer. Because the Delta Sharing protocol emits streamed responses and returns presigned data-file URLs, the proxy must be configured for streaming and its hostname must match the configured Delta Sharing endpoint. This page describes those requirements and the streaming and pagination model behind them.

The Delta Sharing metadata and query endpoints return newline-delimited JSON (application/x-ndjson) that MinIO AIStor streams to the client as it enumerates the files that make up a table. A proxy or load balancer that buffers the full response before forwarding it overflows its buffers or times out on large tables, because the complete response is never held in memory at once on the server side.

Configure the proxy to stream responses and disable response buffering. This is a permanent requirement for Delta Sharing, not a temporary workaround.

location /_delta-sharing/ {
    proxy_pass https://aistor-backend;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_read_timeout 600s;
}

response.streaming.enabled = true
request.streaming.enabled = true

Set MINIO_DELTA_SHARING_ENDPOINT to the externally reachable hostname served by the proxy or load balancer.

MinIO AIStor embeds this value in two places:

• The endpoint field of the generated profile.share, which the client uses for all protocol requests.
• Every presigned data-file URL the client downloads directly.

If the endpoint does not match the proxy hostname, the embedded profile endpoint and presigned URLs do not resolve from the client, or they fail TLS verification. The hostname in MINIO_DELTA_SHARING_ENDPOINT, the certificate the proxy presents, and the address clients connect to must all agree.

Understanding how the protocol moves data explains why the proxy requirements above are mandatory.

The metadata and query endpoints return application/x-ndjson: one JSON object per line, streamed in order. A query response begins with the protocol and table metadata, followed by one file action per data file in the result. Each file action carries a presigned S3 URL for the underlying object.

The client does not stream table rows through the protocol endpoint. Instead, it reads the presigned URLs from the file actions and downloads each data file directly from MinIO AIStor. Presigned URLs expire after a configurable interval, controlled by MINIO_DELTA_SHARING_PRESIGNED_URL_EXPIRY with a default of 1h. Set the expiry long enough that a client finishes downloading all data files for a query before its URLs expire.

A query against a large table can return a large number of file references. The query endpoint accepts a startingToken in the request body, or a pageToken query parameter, and returns a nextPageToken inside the EndStreamAction at the end of a streamed response. The client passes that nextPageToken back as the startingToken on the next request to retrieve the following page of file references.

The discovery endpoints that list shares, schemas, and tables paginate with maxResults and pageToken and return nextPageToken. For the full protocol reference, see Pagination and Query Tuning.

Tune the Delta Sharing rate limits to match the number of concurrent clients and the request volume each generates. The per-token rate limit defaults to 100 requests per second, and the global rate limit defaults to 1000 requests per second across all tokens. A client that exceeds either limit receives an HTTP 429 response with the RESOURCE_EXHAUSTED error code.