Healthcheck Probes

Each AIStor server process exposes unauthenticated endpoints for probing server uptime and deployment high availability for simple healthchecks. These endpoints return an HTTP status code indicating whether the underlying resource is healthy or satisfies read/write quorum. The server exposes no other data through these endpoints.

Use the following endpoint to test if the specified AIStor server is up and ready to serve requests:

curl -I https://aistor.example.net:9000/minio/health/live

Replace https://aistor.example.net:9000 with the DNS hostname and port of the server to check.

A response code of 200 OK indicates the server is online and functional. Any other HTTP codes indicate an issue with reaching the server, such as a transient network issue or potential downtime. Use this endpoint with load balancer healthcheck probes to ensure that client operations route only to healthy nodes.

The healthcheck probe alone cannot determine if the server is offline - only that the current host machine cannot reach the server. Consider configuring a Prometheus alert using the minio_cluster_servers_offline_total metric to detect whether one or more AIStor Server servers are offline.

Use the following endpoint to check if the local AIStor node views the cluster as ‘ready’ to process write operations:

curl -I https://aistor.example.net:9000/minio/health/cluster

Replace https://aistor.example.net:9000 with the DNS hostname and port of any server in the deployment to check. For clusters using a load balancer to manage incoming connections, specify the hostname of the load balancer.

The target node queries its peers for their current drive health and status. A ‘ready’ response indicates that enough peer nodes responded as fully initialized with sufficient healthy drives to support write quorum.

This endpoint alone cannot determine the uptime status of the target node or its peers. For detecting and alerting on node downtime, configure Prometheus alerts using one of the following V3 metrics to detect potential issues or errors on the cluster:

• minio_cluster_servers_offline_total to alert if one or more servers are offline.
• minio_server_drive_free_bytes to alert if the deployment is running low on free drive space.

Use the distributed=true query parameter to verify cluster health from all nodes’ perspectives:

curl -I "https://aistor.example.net:9000/minio/health/cluster?distributed=true"

A distributed readiness check performs a fan-out call to all peer nodes requesting they each perform their own health check. The response then indicates whether all peer nodes agree on cluster readiness, instead of relying on only the single local node’s view.

Combine distributed=true with maintenance=true to verify if a specific node can be safely taken offline while ensuring all other nodes see the cluster as healthy:

curl -I "https://aistor.example.net:9000/minio/health/cluster?distributed=true&maintenance=true"

The endpoint returns one of the following HTTP codes:

The response includes the following headers:

The result of the probe alone does not determine the health of the target server or the cluster’s ability to process operations. It only indicates the local view of the cluster health and status from the targeted node. For example, if the node has a network partition between a peer, the return status reflects that peer as ‘offline’ even if it otherwise is healthy and servicing requests. Similarly, the node may not have fully initialized during the healthcheck while the remainder of the cluster is healthy and operational.

Use the following endpoint to check if the local AIStor node views the cluster as ‘ready’ to process read operations:

curl -I https://aistor.example.net:9000/minio/health/cluster/read

Replace https://aistor.example.net:9000 with the DNS hostname and port of a server in the deployment to check. For clusters using a load balancer to manage incoming connections, specify the hostname of the load balancer.

The target node queries its peers for their current drive health and status. A ‘ready’ response indicates that enough peer nodes responded as fully initialized with healthy drives to support read quorum.

This endpoint alone cannot determine the uptime status of any given peer node. For detecting and alerting on node downtime, configure Prometheus alerts using one of the following metrics to detect potential issues or errors on the cluster:

• minio_cluster_servers_offline_total to alert if one or more servers are offline.
• minio_server_drive_free_bytes to alert if the deployment is running low on free drive space.

The endpoint returns one of the following HTTP codes

The response includes the following headers:

The result of the probe alone does not determine the health of the target server or the cluster’s ability to process operations. It only indicates the local view of the cluster health and status from the targeted node. For example, if the node has a network partition between a peer, the return status reflects that peer as ‘offline’ even if it otherwise is healthy and servicing requests. Similarly, the node may not have fully initialized during the healthcheck while the remainder of the cluster is healthy and operational.

Use the following endpoint to test if an AIStor deployment can maintain both read and write quorum if the target node is taken down for maintenance:

curl -I https://aistor.example.net:9000/minio/health/cluster?maintenance=true

Replace https://aistor.example.net:9000 with the DNS hostname and port of a server in the deployment to check.

The response alone does not indicate the health or availability of the node. It only indicates whether the cluster can tolerate taking the node offline for maintenance operations.

The cluster healthcheck endpoints return 503 Service Unavailable if the target server could not verify the health status of the cluster. This includes scenarios in which the target server has not fully initialized or has encountered an error preventing startup.

The contents of the X-Minio-Server-Status response header contains the specific reason for the failure. Some failures may clear given sufficient time for the server to start up or retry startup operations. If the status persists beyond the normal startup time, check the server logs for errors.

The following table lists status values returned as part of the X-Minio-Server-Status header:

For readiness queries with the distributed flag, the header supports the following additional values:

When a 503 response has no X-Minio-Server-Status header, the cluster does not have sufficient drives online to meet quorum requirements. This occurs when too many drives are offline due to hardware failures, maintenance, or unavailable mount points.

Use mc admin info to identify offline drives. The X-Minio-Healing-Drives response header indicates if healing is in progress on replacement drives. The number of drives required for quorum depends on the erasure code parity configuration.