Storage Hardening

MinIO AIStor supports kernel-level storage protection to prevent accidental deletion of data directories by system administrators while allowing the MinIO AIStor process full control over managed drives. This protection is critical for production environments where a mistyped rm -rf /drive* command could result in catastrophic data loss.

Protection model

Storage hardening provides:

MinIO AIStor process: Full read/write/delete access to managed drives.
Host OS (including root): Cannot delete protected data. With SELinux, other processes have read-only access to protected directories. With eBPF LSM, other processes are blocked from deleting or renaming files owned by a protected UID.
Kernel-level enforcement: Protection cannot be bypassed through normal filesystem operations.
No performance impact: Both implementations add negligible overhead.

Choose an implementation

MinIO AIStor provides two storage protection implementations:

Feature	SELinux	eBPF LSM
Kernel version	Any (built into most enterprise distros)	Linux 5.7 or later, with eBPF LSM enabled
Best for	RHEL, CentOS, Fedora	Ubuntu, Debian, other distros
Configuration	Policy files and labels	Kernel parameter + daemon
Protection model	File labels (`minio_data_t`)	File ownership (UID allowlist)
Maturity	20+ years in production	Newer, but production-ready

Recommendations:

RHEL/CentOS/Fedora: Use SELinux (better integration, native support)
Ubuntu/Debian: Use eBPF LSM (default kernel support, no SELinux required)
Other distributions: Use eBPF LSM for universal compatibility

SELinux-based protection

Use SELinux-based storage protection on RHEL, CentOS, Fedora, and other SELinux-enabled systems.

Prerequisites

RHEL 8/9, CentOS 8/9, Fedora, or any Linux distribution with SELinux.
SELinux in enforcing or permissive mode (not disabled).
SELinux policy development tools.

Check SELinux status:

getenforce

The output should be Enforcing or Permissive. If the output is Disabled, enable SELinux first:

# Edit SELinux config
sudo vi /etc/selinux/config

# Set SELINUX=enforcing
SELINUX=enforcing

# Reboot required
sudo reboot

Install required tools:

sudo dnf install -y policycoreutils-python-utils selinux-policy-devel

Setup

Follow these steps to create and install the SELinux policy.

Create the SELinux policy module

Create a file named minio_storage.te:

policy_module(minio_storage, 1.0.0)

require {
    type unconfined_t;
    type user_t;
    type sysadm_t;
    type init_t;
    class dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
    class file { append create getattr ioctl link lock map open read rename setattr unlink write };
}

# Define MinIO domain
type minio_t;
type minio_exec_t;
type minio_data_t;

# MinIO binary type
files_type(minio_exec_t)

# MinIO data directories type
files_type(minio_data_t)

# Allow MinIO domain to be entered from init
init_daemon_domain(minio_t, minio_exec_t)
domain_type(minio_t)

# MinIO process has FULL access to minio_data_t
allow minio_t minio_data_t:dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
allow minio_t minio_data_t:file { append create getattr ioctl link lock map open read rename setattr unlink write };

# All other processes can ONLY read, CANNOT delete or write
allow unconfined_t minio_data_t:dir { getattr open read search };
allow unconfined_t minio_data_t:file { getattr open read };
allow user_t minio_data_t:dir { getattr open read search };
allow user_t minio_data_t:file { getattr open read };
allow sysadm_t minio_data_t:dir { getattr open read search };
allow sysadm_t minio_data_t:file { getattr open read };

# MinIO network access
corenet_tcp_bind_generic_node(minio_t)
corenet_tcp_connect_http_port(minio_t)
corenet_tcp_bind_http_port(minio_t)
corenet_tcp_bind_generic_port(minio_t)
corenet_udp_bind_generic_port(minio_t)

# MinIO system permissions
kernel_read_system_state(minio_t)
dev_read_urand(minio_t)
fs_getattr_xattr_fs(minio_t)
files_read_etc_files(minio_t)
miscfiles_read_localization(minio_t)
sysnet_dns_name_resolve(minio_t)

# Allow MinIO to execute itself
allow minio_t minio_exec_t:file { execute execute_no_trans getattr map open read };

Compile and install the policy

# Compile and package the policy module with the SELinux devel Makefile
make -f /usr/share/selinux/devel/Makefile minio_storage.pp

# Install the policy module
semodule -i minio_storage.pp

# Verify installation
semodule -l | grep minio_storage

Label the MinIO AIStor binary

# Set file context for MinIO AIStor binary
semanage fcontext -a -t minio_exec_t "/usr/local/bin/minio"

# Apply the label
restorecon -v /usr/local/bin/minio

# Verify the label
ls -Z /usr/local/bin/minio

The output should show system_u:object_r:minio_exec_t:s0.

Label data directories

For each drive that MinIO AIStor manages:

# Add file context rules for each drive
semanage fcontext -a -t minio_data_t "/drive1(/.*)?"
semanage fcontext -a -t minio_data_t "/drive2(/.*)?"
semanage fcontext -a -t minio_data_t "/drive3(/.*)?"
semanage fcontext -a -t minio_data_t "/drive4(/.*)?"

# Apply labels recursively
restorecon -R -v /drive1 /drive2 /drive3 /drive4

# Verify labels
ls -Z /drive1

The output should show system_u:object_r:minio_data_t:s0.

Verify SELinux protection

Test that protection is working:

# Write a test object through MinIO AIStor
mc cp /etc/hostname ALIAS/BUCKET/test-file

# Try to delete the object's data on the drive as root (should fail)
sudo rm -r /drive1/BUCKET/test-file
# Expected: rm: cannot remove '/drive1/...': Permission denied

Check that processes are running in the correct SELinux domain:

# Check MinIO AIStor process context
ps -eZ | grep minio
# Expected: system_u:system_r:minio_t:s0

# Check file contexts
ls -Z /drive1
# Expected: system_u:object_r:minio_data_t:s0

eBPF LSM-based protection

Use eBPF LSM-based storage protection on Ubuntu, Debian, and other distributions without SELinux.

Prerequisites

AIStor-compatible Linux kernel with eBPF LSM support.
CONFIG_BPF_LSM=y kernel configuration.
lsm=...,bpf kernel boot parameter.
Root access for installation.

Check kernel support:

# Check kernel version
uname -r

# Check if eBPF LSM is enabled
cat /sys/kernel/security/lsm | grep -q bpf && echo "Supported" || echo "Not supported"

# Check kernel configuration (if available)
grep CONFIG_BPF_LSM /boot/config-$(uname -r)

Enable eBPF LSM

If eBPF LSM is not in the LSM list, enable it via kernel boot parameters.

# Edit GRUB configuration
sudo vi /etc/default/grub

# Add or modify GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash lsm=lockdown,yama,integrity,apparmor,bpf"

# Update GRUB
sudo update-grub

# Reboot
sudo reboot

# Edit GRUB configuration
sudo vi /etc/default/grub

# Add or modify GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="... lsm=selinux,bpf"

# Update GRUB
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Reboot
sudo reboot

Install required packages

sudo apt-get update
sudo apt-get install -y \
    clang \
    llvm \
    libbpf-dev \
    linux-tools-generic \
    linux-headers-$(uname -r) \
    golang-go

sudo dnf install -y \
    clang \
    llvm \
    libbpf-devel \
    bpftool \
    kernel-devel \
    golang

How eBPF protection works

The minio-lsm-bpf daemon uses a pure UID-based protection model. You pass it one or more UIDs to protect using the -u option, typically the UID of the user that runs MinIO AIStor. The daemon then protects all files owned by those UIDs, regardless of their location on disk:

The daemon attaches eBPF programs to the kernel LSM hooks.
When a process attempts to delete a file, the hook checks whether the file is owned by a protected UID.
If the file is owned by a protected UID, the operation is allowed only when the deleting process runs as the same UID.
All other processes, including root, are blocked from deleting files owned by a protected UID.

The eBPF LSM hooks intercept:

unlink() - File deletion.
rmdir() - Directory deletion.
rename() - Moving or renaming files.

Because protection is based on file ownership rather than path, any drive whose files are owned by a protected UID is covered automatically.

Setup

Follow these steps to build, install, and run the protection daemon from the scripts/ebpf directory of the MinIO AIStor source tree.

Verify kernel support

# Check eBPF LSM support
if ! cat /sys/kernel/security/lsm | grep -q bpf; then
    echo "eBPF LSM not enabled - add 'bpf' to kernel boot parameters"
    exit 1
fi

echo "Kernel support: OK"

You can also run make check-kernel from the scripts/ebpf directory to verify that the system is ready.

Build the eBPF program

cd scripts/ebpf

# Verify the required build tools are present
make check-tools

# Build the binary (default 'all' target)
make

# Verify the binary was created
ls -lh minio-lsm-bpf

Install the binary

# Install to system path
sudo install -m 0755 minio-lsm-bpf /usr/local/bin/

Create the systemd service

Determine the UID that MinIO AIStor runs as:

# Get the UID of minio-user (for example, 1000)
id -u minio-user

Create /etc/systemd/system/minio-lsm-bpf.service, replacing 1000 with the UID from the previous command:

[Unit]
Description=MinIO eBPF LSM Protection
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/minio-lsm-bpf -u 1000
Restart=always
RestartSec=10
LimitMEMLOCK=infinity
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

To protect more than one UID, repeat the -u option, for example ExecStart=/usr/local/bin/minio-lsm-bpf -u 1000 -u 1001.

The daemon supports the following options:

-u UID - UID whose files are protected (required, can be repeated).
-v - Enable verbose logging (logs allowed operations in addition to blocked operations).

Enable and start the service

sudo systemctl daemon-reload
sudo systemctl enable --now minio-lsm-bpf
sudo systemctl status minio-lsm-bpf

Verify eBPF protection

Test that protection is working:

# Create a test file owned by the protected user
sudo -u minio-user touch /drive1/test-file

# Try to delete as root (should fail)
sudo rm /drive1/test-file
# Expected: rm: cannot remove '/drive1/test-file': Operation not permitted

# Verify the file still exists
ls -l /drive1/test-file

# Delete as the file owner (should succeed)
sudo -u minio-user rm /drive1/test-file

Check service status:

# Service status
sudo systemctl status minio-lsm-bpf

# View logs (blocked operations appear in real time)
sudo journalctl -u minio-lsm-bpf -f
# Example output:
# BLOCKED unlink: op_pid=1234 op_uid=0 comm=rm owner_uid=1000

Troubleshooting

MinIO AIStor cannot start

If MinIO AIStor fails to start after enabling protection:

# Check SELinux denials
sudo ausearch -m avc -ts recent

# Check MinIO AIStor binary label
ls -Z /usr/local/bin/minio

# Temporarily set permissive mode to diagnose
sudo setenforce 0
# Start MinIO AIStor, then check audit log
sudo ausearch -m avc -ts recent | audit2allow

# Check if the minio-lsm-bpf service is running
sudo systemctl status minio-lsm-bpf

# Check logs for errors
sudo journalctl -u minio-lsm-bpf -n 50

# Verify kernel support
cat /sys/kernel/security/lsm | grep -q bpf && echo "Supported" || echo "Not supported"

# Restart protection service
sudo systemctl restart minio-lsm-bpf

MinIO AIStor cannot write to drives

If MinIO AIStor cannot write to protected drives:

# Verify directory labels
ls -Z /drive1

# Relabel if needed
restorecon -R -v /drive1

# Check for context mismatches
sudo ausearch -m avc -ts recent | grep minio

The daemon only blocks deletions of files owned by a protected UID; it never blocks the owning UID from writing or deleting its own files. If MinIO AIStor cannot write to a drive, confirm that the drive’s files are owned by the UID that MinIO AIStor runs as and that this UID matches the one passed with -u.

# Confirm the protected UID matches the MinIO AIStor user
id -u minio-user
sudo systemctl show minio-lsm-bpf -p ExecStart

# Confirm drive ownership
ls -ln /drive1

# Check logs for blocked operations
sudo journalctl -u minio-lsm-bpf -n 50

Protection not blocking root

If root can still delete files:

# Verify SELinux is enforcing
getenforce

# Check if policy is loaded
semodule -l | grep minio_storage

# Verify file contexts
ls -Z /drive1

# Verify eBPF program is loaded
sudo bpftool prog list | grep minio

# Verify LSM hooks are attached
sudo bpftool link list

# Check the service logs for blocked operations
sudo journalctl -u minio-lsm-bpf | grep BLOCKED

Security considerations

Defense in depth

Storage hardening is one layer of defense. Implement additional measures:

Regular backups to separate storage
Filesystem snapshots (ZFS, Btrfs, LVM)
Restricted SSH access with key-based authentication
Multi-factor authentication for administrative access
Monitoring for unauthorized access attempts

Limitations

Both implementations have known limitations:

Limitation	Mitigation
Root can disable the protection service	Monitor service status, use systemd protections
Root can change kernel parameters via reboot	Use TPM/UEFI Secure Boot to protect boot parameters
Direct block device access bypasses filesystem	Use disk encryption, monitor block device access

Adding new drives

When adding new drives, label them with the correct SELinux context:

semanage fcontext -a -t minio_data_t "/drive9(/.*)?"
restorecon -R -v /drive9

eBPF protection is based on file ownership, not path. Any drive whose files are owned by a protected UID is covered automatically, regardless of its mount path. No configuration changes are needed when adding a drive, provided its files are owned by an already-protected UID.

To protect files owned by an additional UID, add another -u UID argument to the daemon’s ExecStart line and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart minio-lsm-bpf

Uninstalling

# Remove the policy module
sudo semodule -r minio_storage

# Remove file context rules
sudo semanage fcontext -d "/usr/local/bin/minio"
sudo semanage fcontext -d "/drive1(/.*)?"
# Repeat for other drives

# Restore default labels
sudo restorecon -R -v /usr/local/bin/minio /drive1

sudo systemctl disable --now minio-lsm-bpf
sudo rm /etc/systemd/system/minio-lsm-bpf.service
sudo systemctl daemon-reload
sudo rm /usr/local/bin/minio-lsm-bpf

Storage Hardening

On this page