Bucket-level Quality of Service

Quality of Service (QoS) applies rate limiting and concurrency controls to S3, SFTP, and FTP requests on a per-bucket basis. Use it to control API request rates and concurrent operations, so that no single workload overloads the cluster.

QoS has two enforcement layers

API QoS controls requests — requests per second (rps) and concurrent operations (concurrency) — inside the MinIO AIStor server. API QoS does not cap bandwidth.

QoS also has a network layer, implemented by minwall, that runs in front of (or co-located with) each MinIO AIStor server. Network QoS is the only place that caps bandwidth (bytes per second), because bandwidth must be shaped before the storage process accepts the request body or begins streaming the response — once the connection is past TLS, signature verification, and the erasure-coded I/O pipeline, the cost has already been paid. Network QoS can also enforce per-API request-rate caps that reject traffic before authentication.

See Network QoS for the full rationale, schema, and examples.

You can configure rules with QoS to do the following:

Feature Description
Rate Limiting Control the number of requests per second for specified operations
Concurrency Control Limit the number of concurrent operations
Prefix-based Targeting Apply limits to specific object prefixes within buckets
API-specific Rules Target specific S3 operations
Priority-based Evaluation Use rule priorities for complex scenarios

In production, write your QoS rules to a YAML configuration file:

version: "v1" 
rules:
  - id: "upload-rate-limit"
    label: "Limit upload operations"
    priority: 1
    objectPrefix: "uploads/"
    api: "s3.PutObject"
    rate: 100
    burst: 20
    limit: "rps"

Then apply these rules with the following command:

mc qos rule import ALIAS/my-bucket < /path/to/qos-config.yaml

To update rules, edit the YAML file and run mc qos rule import again. The YAML is replaced, not appended, so make sure not to remove or change any existing rules you need to keep.

See also QoS Rule Examples.

You can also add individual rules with the mc qos rule add command:

mc qos rule add ALIAS/my-bucket/my-prefix  --api "s3.GetObject" --rate 20   --limit rps --burst 50 --priority 1

See also the reference documentation for mc qos rule.

You can also monitor operations configured with QoS with mc qos status.

QoS rule syntax

Parameters

Parameter Type Description
objectPrefix string Required
Object prefix to match.
"" to match all objects. Wildcards are not allowed.
priority integer Required
Rule priority.
The lowest number has the highest priority.
api string Required
API operation name.
Must be a valid S3, SFTP, or FTP API name. See Supported APIs for QoS.
Accepts the method wildcards s3.GET*, s3.PUT*, s3.LIST*, s3.DELETE*, s3.HEAD*, and s3.COPY*.
rate integer Required
Rate limit value. Must be greater than 0.
burst integer Required  if limit is "rps". Must be greater than 0.
Number of requests to temporarily allow above the rate.
id string Optional
Identifier for the rule.
Default value: a generated UUID.
Should be unique across all rules.
label string Optional
Human-readable description
limit string Optional
Limit type. One of "rps" or "concurrency".
Default value "rps"

The combination of objectPrefix and api must be unique across all rules.

Duplicate ID values
If you provide duplicate ID values, all rules with the same ID are applied, but if you call mc qos rule remove, all rules with the same ID are deleted.

Limit types

MinIO AIStor QoS supports the following limit types:

  • "rps" (Requests Per Second)

    Controls the rate of API requests using a token bucket algorithm.

    YAML example:

    limit: "rps"
rate: 100      # 100 requests per second
burst: 20      # Allow bursts up to 20 requests above the rate

  • "concurrency"

    Controls the maximum number of concurrent operations. When rate in-flight requests are already running, AIStor Server rejects further requests with 429 (Too Many Requests) rather than queuing them.

    YAML example:

    limit: "concurrency"
rate: 10       # Maximum 10 concurrent operations

Rule evaluation order

AIStor Server selects at most one rule per request, in two stages.

First it selects by API name. It looks for rules that name the request’s API exactly, such as s3.PutObject. Only when no exact-name rule matches does it fall back to the matching method wildcard, such as s3.PUT*. An exact API name therefore takes precedence over a wildcard regardless of priority: a priority: 9 rule for s3.PutObject wins over a priority: 1 rule for s3.PUT*.

Then, among the rules for the selected API name, it takes the first one whose objectPrefix matches the request, in priority order, where the lowest number has the highest priority.

Priority therefore orders rules within a single API name. It does not order rules across different API names.

Give overlapping rules distinct priorities
Two rules for the same API name can both match a request, for example objectPrefix: "critical/" and objectPrefix: "". When such rules share the same priority, the order in which AIStor Server evaluates them is not defined, so which one applies is not predictable. Assign a distinct priority to every rule that can match the same request.

A duplicate combination of object prefix and API operation fails with an error that describes the issue. Duplicate priority values are accepted.

If no rule matches, the request proceeds with no QoS limits.

Troubleshooting

Requests being throttled unexpectedly (429 / Too Many Requests)

When a request exceeds a QoS rate or concurrency limit, AIStor rejects it with an HTTP 429 (Too Many Requests) response.

AIStor returns an X-Minio-QOS-Rule-Meta response header that identifies the QoS rule applied to the request. The header appears on the 429 response when a rule throttles the request, and also on a successful response whenever a rule matched. Inspect it to pinpoint the responsible rule, then review that rule’s priority, matching criteria, and rate or burst settings.

To diagnose unexpected throttling:

  1. Check QoS metrics for active throttling with mc qos status.
  2. Inspect the X-Minio-QOS-Rule-Meta header on 429 responses to identify which rule matched the request.
  3. Verify rule priorities and matching logic, including object prefix and API operation matches. See Rule evaluation order.
  4. Review the rate and burst settings for the responsible rule.
  5. Compare actual request patterns against the configured limits.