Hardware and System Requirements

This section provides guidance around recommended hardware configurations in production AIStor Server deployments.

The following checklist follows our Recommended Configuration for each physical or virtual host supporting the AIStor Server deployment. Use the Erasure Code Calculator to provide baseline guidance for the number of hosts and drives necessary to meet your storage and availability requirements.

The following areas have the greatest impact on AIStor Server performance, listed in order of importance:

The recommendations above reflect our experience with assisting enterprise customers in deploying on a variety of IT infrastructures while maintaining the desired SLA/SLO. While AIStor Server may run on less than the minimum recommended topology, any potential cost savings come at the risk of decreased reliability, performance, or overall functionality.

AIStor Server recommends high speed networking to support the maximum possible throughput of the attached storage (aggregated drives, storage controllers, and PCIe busses). The following table provides a general guideline for the maximum storage throughput supported by a given physical or virtual network interface. This table assumes all network infrastructure components, such as routers, switches, and physical cabling, also supports the NIC bandwidth.

Networking has the greatest impact on AIStor Server performance, where low per-host bandwidth artificially constrains the potential performance of the storage. If the node host can only support 25GbE of throughput, then the read/write performance of that host can never exceed that amount regardless of the number or speed of attached drives.

Memory primarily constrains the number of concurrent simultaneous connections per node. For deployments with caching enabled, the AIStor Server allocates up to 50% of host memory as a drive-level per-shard cache for supporting GET operations.

You can calculate the maximum number of concurrent requests per node with this formula:

To calculate the amount of RAM used for each request, use this formula:

10MiB is the default erasure block size v1. 1 MiB is the default erasure block size v2.

The following table lists the maximum concurrent requests on a node based on the number of host drives and the free system RAM:

AIStor recommends using flash-based storage (NVMe or SSD) for all workload types at any scale. Workloads that require high performance should prefer NVMe over SSD.

For best performance, provide AIStor with drives attached directly to each host node as a JBOD array with no RAID or similar management layers. AIStor strongly recommends the XFS filesystem for best performance and behavior at scale. Using any other type of backing storage (SAN/NAS, ext4, RAID, LVM) typically results in a reduction in performance, reliability, predictability, and consistency.

For Kubernetes deployments, utilize a Container Storage Interface (CSI) such as AIStor Drive Manager (DirectPV) designed to manage and provision Persistent Volumes on the same host as the AIStor Server pod. Where possible, configure the CSI to format drives as XFS according to the guidelines provided in this documentation.

Ensure all drives in a given AIStor Server pool have the same capacity, make, and model. Mixing drive types typically results in performance degradation, as the slowest or smallest drives in the deployment become a bottleneck regardless of the capabilities of larger or faster drives.

The AIStor Server process requires that drives maintain their ordering at the mounted position across restarts. Arbitrary migration of a drive with existing AIStor Server data to a new mount position, whether intentional or as the result of OS-level behavior, may result in unexpected behavior including data loss.

For Kubernetes environments, your preferred CSI should provide the necessary guarantees around drive mounting and path preservation across host or pod restarts.

On Baremetal environments, you must use /etc/fstab or a similar mount control system to mount drives at a consistent path. For example:

$ nano /etc/fstab

# <file system>        <mount point>    <type>  <options>         <dump>  <pass>
LABEL=MINIODRIVE1      /mnt/drive-1     xfs     defaults,noatime  0       2
LABEL=MINIODRIVE2      /mnt/drive-2     xfs     defaults,noatime  0       2
LABEL=MINIODRIVE3      /mnt/drive-3     xfs     defaults,noatime  0       2
LABEL=MINIODRIVE4      /mnt/drive-4     xfs     defaults,noatime  0       2

You can use mount -a to mount those drives at those paths during initial setup. The Operating System should otherwise mount these drives as part of the node startup process.

Use label-based mounting rules over UUID-based rules. Label-based rules allow swapping an unhealthy or non-working drive with a replacement that has matching format and label. UUID-based rules require editing the /etc/fstab file to replace mappings with the new drive UUID.

The following subsections apply to XFS formatted drives.

For Kubernetes deployments, defer to your choice of CSI or Persistent Volume type for support in setting these configurations.

Disable retry-on-error behavior using the max_retries configuration for the following error classes:

• EIO Error when reading or writing
• ENOSPC Error no space left on device
• default All other errors

The default max_retries setting typically directs the filesystem to retry-on-error indefinitely instead of propagating the error. Each AIStor Server process can handle XFS errors appropriately, such that the retry-on-error behavior at best introduces at most unnecessary latency or performance degradation.

The following script iterates through all drives at the specified mount path and sets the XFS max_retries setting to 0 or "fail immediately on error" for the recommended error classes. The script ignores any drives not mounted, either manually or through /etc/fstab. Modify the /mnt/drive line to match the pattern used for your AIStor Server drives.

#!/bin/bash

for i in $(df -h | grep /mnt/drive | awk '{ print $1 }'); do
      mountPath="$(df -h | grep $i | awk '{ print $6 }')"
      deviceName="$(basename $i)"
      echo "Modifying xfs max_retries and retry_timeout_seconds for drive $i mounted at $mountPath"
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/EIO/max_retries
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/ENOSPC/max_retries
      echo 0 > /sys/fs/xfs/$deviceName/error/metadata/default/max_retries
done
exit 0

You must run this script on all AIStor Server hosts and configure the script to re-run on reboot, as Linux Operating Systems do not typically persist these changes. You can use a cron job with the @reboot timing to run the above script whenever the node restarts and ensure all drives have retry-on-error disabled. Use crontab -e to create the following job, modifying the script path to match that on each node:

@reboot /opt/minio/xfs-retry-settings.sh