Deploy the RDMA server

Install the RDMA build of AIStor Server and configure the service to run it.

Complete Configure the RDMA fabric first. An AIStor node with the RDMA build installed but an unconfigured fabric serves every request over TCP.

Each release publishes the RDMA build for Linux on amd64 and arm64, alongside the standard artifacts.

The binary carries .asc, .minisig, and .sha256sum files for verification, the same as the standard build.

The package architecture suffix follows each package format’s own convention:

The RDMA package installs the RDMA binary and its shared libraries. It does not install the systemd unit, the minio-user account, or the configuration files, so install the standard AIStor package first and add the RDMA package on top.

Install both on every node:

VERSION=20260807183435.0.0
ARCH=$(dpkg --print-architecture)

wget https://dl.min.io/aistor/minio/release/linux-${ARCH}/aistor.deb
wget https://dl.min.io/aistor/minio/release/linux-${ARCH}/minio.rdma_${VERSION}_${ARCH}.deb

sudo dpkg -i aistor.deb
sudo apt-get install -f -y
sudo dpkg -i minio.rdma_${VERSION}_${ARCH}.deb
sudo apt-get install -f -y

VERSION=20260807183435.0.0
case "$(uname -m)" in
  x86_64)  DL_ARCH=linux-amd64; PKG_ARCH=x86_64 ;;
  aarch64) DL_ARCH=linux-arm64; PKG_ARCH=aarch64 ;;
esac

wget https://dl.min.io/aistor/minio/release/${DL_ARCH}/aistor.rpm
wget https://dl.min.io/aistor/minio/release/${DL_ARCH}/minio.rdma-${VERSION}-1.${PKG_ARCH}.rpm

sudo dnf install -y aistor.rpm
sudo dnf install -y minio.rdma-${VERSION}-1.${PKG_ARCH}.rpm

Installing with apt-get install -f or dnf install rather than dpkg -i or rpm -i alone lets the package manager pull the RDMA runtime dependencies.

The package’s post-install script writes the ld.so.conf.d entry and runs ldconfig, so the libraries resolve without further action.

It also pulls these runtime dependencies:

Confirm the binary resolves its libraries:

ldd /usr/local/bin/minio.rdma | grep -E "s3rdma|p2p_rdma|ibverbs|rdmacm"

Every line must show a resolved path. A not found entry means ldconfig did not run; rerun sudo ldconfig and check again.

The standard package installs /lib/systemd/system/minio.service, which runs /usr/local/bin/minio. That path is a symlink to the standard, non-RDMA binary.

Create the drop-in on every node:

sudo mkdir -p /etc/systemd/system/minio.service.d

sudo tee /etc/systemd/system/minio.service.d/rdma.conf >/dev/null <<'EOF'
[Unit]
# AssertFileIsExecutable belongs to [Unit], not [Service]. The empty assignment
# clears the packaged value before the new one is set.
AssertFileIsExecutable=
AssertFileIsExecutable=/usr/local/bin/minio.rdma

[Service]
# Run the RDMA build instead of the standard binary.
ExecStart=
ExecStart=/usr/local/bin/minio.rdma server $MINIO_OPTS $MINIO_VOLUMES

# RDMA pins its transfer buffers in physical memory. See "Raise the locked
# memory limit" for how to size a finite value instead.
LimitMEMLOCK=infinity
EOF

sudo systemctl daemon-reload

The empty ExecStart= and AssertFileIsExecutable= lines are not optional. Both directives accumulate entries across drop-ins. Without the ExecStart= reset, systemd refuses the unit with Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

AssertFileIsExecutable must sit under [Unit]. Placing it under [Service] produces Unknown key 'AssertFileIsExecutable' in section [Service], ignoring and leaves the packaged assertion pointing at the standard binary.

Check the drop-in parses:

systemd-analyze verify minio.service

Confirm the merged unit before restarting:

systemctl cat minio

The output shows the packaged unit followed by the drop-in. Check that the final ExecStart names minio.rdma.

RDMA registers its transfer buffers with the NIC, which locks them into physical memory. The packaged unit sets no LimitMEMLOCK, so the service inherits the systemd default of 8 MiB. RDMA needs several orders of magnitude more.

Setting /etc/security/limits.conf does not fix this. That file is read by PAM during login, and a systemd-managed service never goes through PAM. The limit must come from the unit, which is why the drop-in above sets it.

The capabilities the packaged unit already grants do not help either. Only CAP_IPC_LOCK exempts a process from RLIMIT_MEMLOCK, and the unit grants CAP_NET_BIND_SERVICE, CAP_SYS_ADMIN, and CAP_DAC_OVERRIDE. Raise the limit rather than adding the capability.

An RDMA node registers memory in two places.

The inter-node receive-buffer pool. Its size is the product of two settings:

At the defaults that is 16 GiB, pinned at startup.

The transfer buffer pools. The RDMA build allocates three pools of 4 MiB buffers at startup, sized from the node’s role: 2048 buffers each on a distributed deployment, 1024 on a single node. That is roughly 24 GiB on a distributed node and 12 GiB on a single node.

Together, a distributed node at default settings registers roughly 40 GiB. LimitMEMLOCK=infinity avoids having to track this as the settings change.

If you set a finite value, cover both figures and round up generously.

On a memory-constrained node, lower MINIO_RDMA_POOL_COUNT, or lower MINIO_RDMA_POOL_BUF_SIZE and raise the count to compensate.

Lowering the slab size has a throughput cost. A PUT shard larger than one slab falls back to TCP rather than using the pre-registered pool. The 64 MiB default is sized for the erasure shard of a typical 64 MiB object.

The two pools fail differently, and neither stops the server from starting.

An under-sized limit that fails the receive pool leaves inter-node RDMA off. The server logs the failure and serves every request over TCP.

An under-sized limit that fails the transfer-buffer registration leaves RDMA running in a degraded mode, registering memory on every write instead of reusing pre-registered regions:

RDMA: sender buffer registration failed (falling back to per-write reg): <error>

Neither case is an error the server surfaces to clients, which is why the checks in Validate the RDMA deployment matter.

The packaged unit reads /etc/default/minio as its EnvironmentFile, and that file points MINIO_CONFIG_ENV_FILE at /etc/minio/config.env.

Put every MINIO_RDMA_* variable in /etc/default/minio.

Several of them, including MINIO_RDMA_POOL_COUNT, MINIO_RDMA_POOL_BUF_SIZE, MINIO_RDMA_MIN_SIZE, and MINIO_RDMA_CONCURRENCY, are read during package initialization, before the server loads config.env. Placed in config.env, they are read too late and silently have no effect. Keeping all of them in one file avoids having to track which are which.

Enable inter-node RDMA on every node:

sudo tee -a /etc/default/minio >/dev/null <<'EOF'

# Inter-node RDMA
MINIO_RDMA_INTERNODE=on
EOF

The settings must be identical across every node in the deployment. Verify with a checksum:

shasum /etc/default/minio

S3 over RDMA needs no setting. The client sends the x-amz-rdma-token header, and the server either serves the request over RDMA or declines it and the client falls back to HTTP.

For the full list of settings, see RDMA settings.

Inter-node RDMA listens on TCP port 5555 by default, configurable with MINIO_RDMA_PEER_PORT. Open it between cluster nodes only.

Restrict the rule to the cluster’s own addresses. The RDMA transport does not authenticate source addresses, so an unrestricted rule exposes the peer port to anything that can route to the node. Replace 10.0.0.0/24 below with the subnet carrying your inter-node traffic.

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" port port="5555" protocol="tcp" accept'
sudo firewall-cmd --reload

sudo iptables -A INPUT -p tcp --dport 5555 -s 10.0.0.0/24 -j ACCEPT

# Debian, Ubuntu
sudo apt-get install -y iptables-persistent
sudo netfilter-persistent save

# RHEL, Rocky, AlmaLinux
sudo dnf install -y iptables-services
sudo systemctl enable --now iptables
sudo service iptables save

See Security considerations.

Restart the service on every node:

sudo systemctl restart minio

Then confirm the deployment is carrying RDMA traffic. See Validate the RDMA deployment.

The standard package and the RDMA package upgrade independently, and nothing prevents them from drifting to different versions.

Upgrade both together, and confirm the versions match afterward:

dpkg-query -W -f='${Package} ${Version}\n' aistor minio.rdma

rpm -q --qf '%{NAME} %{VERSION}\n' aistor minio.rdma

The systemd drop-in and the ld.so.conf.d entry both survive package upgrades. Re-run systemctl cat minio after an upgrade to confirm the drop-in is still merged.

RDMA transfers are not encrypted. The transport assumes physical-layer security.

• Put RDMA traffic on a dedicated VLAN, separate from client traffic.
• Restrict the peer port to cluster nodes only.
• The RDMA transport does not authenticate source addresses. SourceIp is an S3 policy condition, so restrict requests by source address through bucket or user policy, which the standard S3 authorization path evaluates.

S3-over-RDMA requests are authenticated through the standard S3 authentication path, the same as HTTP requests. Peer RDMA connections require successful Grid RPC authentication.

Each node maintains isolated memory registration contexts, and memory regions are registered with specific access rights. No direct cross-node memory access is permitted.