Storage Hardening

MinIO AIStor supports kernel-level storage protection to prevent accidental deletion of data directories by system administrators while allowing the MinIO AIStor process full control over managed drives. This protection is critical for production environments where a mistyped rm -rf /drive* command could result in catastrophic data loss.

Storage hardening provides:

• MinIO AIStor process: Full read/write/delete access to managed drives.
• Host OS (including root): Read-only access to protected directories.
• Kernel-level enforcement: Protection cannot be bypassed through normal filesystem operations.
• No performance impact: Both implementations add negligible overhead.

MinIO AIStor provides two storage protection implementations:

Recommendations:

• RHEL/CentOS/Fedora: Use SELinux (better integration, native support)
• Ubuntu/Debian: Use eBPF LSM (default kernel support, no SELinux required)
• Other distributions: Use eBPF LSM for universal compatibility

Use SELinux-based storage protection on RHEL, CentOS, Fedora, and other SELinux-enabled systems.

• RHEL 8/9, CentOS 8/9, Fedora, or any Linux distribution with SELinux.
• SELinux in enforcing or permissive mode (not disabled).
• SELinux policy development tools.

Check SELinux status:

getenforce

The output should be Enforcing or Permissive. If the output is Disabled, enable SELinux first:

# Edit SELinux config
sudo vi /etc/selinux/config

# Set SELINUX=enforcing
SELINUX=enforcing

# Reboot required
sudo reboot

Install required tools:

sudo dnf install -y policycoreutils-python-utils selinux-policy-devel

Use the setup script for automatic installation:

curl -O https://raw.githubusercontent.com/miniohq/eos/master/scripts/setup-selinux-protection.sh
chmod +x setup-selinux-protection.sh
sudo ./setup-selinux-protection.sh

The script performs the following actions:

1. Verifies SELinux is enabled.
2. Installs required packages.
3. Creates and compiles the SELinux policy module.
4. Labels the MinIO AIStor binary and data directories.
5. Creates a hardened systemd service.
6. Validates the protection is working.

For manual installation or customization, follow these steps.

Create a file named minio_storage.te:

policy_module(minio_storage, 1.0.0)

require {
    type unconfined_t;
    type user_t;
    type sysadm_t;
    type init_t;
    class dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
    class file { append create getattr ioctl link lock map open read rename setattr unlink write };
}

# Define MinIO domain
type minio_t;
type minio_exec_t;
type minio_data_t;

# MinIO binary type
files_type(minio_exec_t)

# MinIO data directories type
files_type(minio_data_t)

# Allow MinIO domain to be entered from init
init_daemon_domain(minio_t, minio_exec_t)
domain_type(minio_t)

# MinIO process has FULL access to minio_data_t
allow minio_t minio_data_t:dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
allow minio_t minio_data_t:file { append create getattr ioctl link lock map open read rename setattr unlink write };

# All other processes can ONLY read, CANNOT delete or write
allow unconfined_t minio_data_t:dir { getattr open read search };
allow unconfined_t minio_data_t:file { getattr open read };
allow user_t minio_data_t:dir { getattr open read search };
allow user_t minio_data_t:file { getattr open read };
allow sysadm_t minio_data_t:dir { getattr open read search };
allow sysadm_t minio_data_t:file { getattr open read };

# MinIO network access
corenet_tcp_bind_generic_node(minio_t)
corenet_tcp_connect_http_port(minio_t)
corenet_tcp_bind_http_port(minio_t)
corenet_tcp_bind_generic_port(minio_t)
corenet_udp_bind_generic_port(minio_t)

# MinIO system permissions
kernel_read_system_state(minio_t)
dev_read_urand(minio_t)
fs_getattr_xattr_fs(minio_t)
files_read_etc_files(minio_t)
miscfiles_read_localization(minio_t)
sysnet_dns_name_resolve(minio_t)

# Allow MinIO to execute itself
allow minio_t minio_exec_t:file { execute execute_no_trans getattr map open read };

# Compile the policy module
checkmodule -M -m -o minio_storage.mod minio_storage.te

# Package the module
semodule_package -o minio_storage.pp -m minio_storage.mod

# Install the policy module
semodule -i minio_storage.pp

# Verify installation
semodule -l | grep minio_storage

# Set file context for MinIO AIStor binary
semanage fcontext -a -t minio_exec_t "/usr/local/bin/minio"

# Apply the label
restorecon -v /usr/local/bin/minio

# Verify the label
ls -Z /usr/local/bin/minio

The output should show system_u:object_r:minio_exec_t:s0.

For each drive that MinIO AIStor manages:

# Add file context rules for each drive
semanage fcontext -a -t minio_data_t "/drive1(/.*)?"
semanage fcontext -a -t minio_data_t "/drive2(/.*)?"
semanage fcontext -a -t minio_data_t "/drive3(/.*)?"
semanage fcontext -a -t minio_data_t "/drive4(/.*)?"

# Apply labels recursively
restorecon -R -v /drive1 /drive2 /drive3 /drive4

# Verify labels
ls -Z /drive1

The output should show system_u:object_r:minio_data_t:s0.

Test that protection is working:

# Create a test file (as MinIO AIStor or with proper context)
sudo -u minio-user touch /drive1/test-file

# Try to delete as root (should fail)
sudo rm /drive1/test-file
# Expected: rm: cannot remove '/drive1/...': Permission denied

Check that processes are running in the correct SELinux domain:

# Check MinIO AIStor process context
ps -eZ | grep minio
# Expected: system_u:system_r:minio_t:s0

# Check file contexts
ls -Z /drive1
# Expected: system_u:object_r:minio_data_t:s0

Use eBPF LSM-based storage protection on Ubuntu, Debian, and other distributions without SELinux.

• AIStor-compatible Linux kernel with eBPF LSM support.
• CONFIG_BPF_LSM=y kernel configuration.
• lsm=...,bpf kernel boot parameter.
• Root access for installation.

Check kernel support:

# Check kernel version
uname -r

# Check if eBPF LSM is enabled
cat /sys/kernel/security/lsm | grep -q bpf && echo "Supported" || echo "Not supported"

# Check kernel configuration (if available)
grep CONFIG_BPF_LSM /boot/config-$(uname -r)

If eBPF LSM is not in the LSM list, enable it via kernel boot parameters.

# Edit GRUB configuration
sudo vi /etc/default/grub

# Add or modify GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash lsm=lockdown,yama,integrity,apparmor,bpf"

# Update GRUB
sudo update-grub

# Reboot
sudo reboot

# Edit GRUB configuration
sudo vi /etc/default/grub

# Add or modify GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="... lsm=selinux,bpf"

# Update GRUB
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Reboot
sudo reboot

sudo apt-get update
sudo apt-get install -y \
    clang \
    llvm \
    libbpf-dev \
    linux-tools-generic \
    linux-headers-$(uname -r) \
    golang-go

sudo dnf install -y \
    clang \
    llvm \
    libbpf-devel \
    bpftool \
    kernel-devel \
    golang

Use the setup script for automatic installation:

curl -O https://raw.githubusercontent.com/miniohq/eos/master/scripts/setup-ebpf-protection.sh
chmod +x setup-ebpf-protection.sh
sudo ./setup-ebpf-protection.sh

The script performs the following actions:

1. Checks kernel support for eBPF LSM.
2. Installs required dependencies.
3. Builds the eBPF program and userspace loader.
4. Installs the binary to /usr/local/bin/minio-protect.
5. Creates and starts a systemd service.
6. Validates the installation.

For manual installation or customization, follow these steps.

# Check eBPF LSM support
if ! cat /sys/kernel/security/lsm | grep -q bpf; then
    echo "eBPF LSM not enabled - add 'bpf' to kernel boot parameters"
    exit 1
fi

echo "Kernel support: OK"

cd scripts/ebpf

# Generate vmlinux.h (kernel data structures)
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

# Build using Makefile
make clean build

# Verify binary was created
ls -lh minio-protect

# Install to system path
sudo install -m 755 minio-protect /usr/local/bin/

# Test installation
/usr/local/bin/minio-protect check

Create /etc/systemd/system/minio-protect.service:

[Unit]
Description=MinIO AIStor Storage Protection (eBPF)
Documentation=man:minio-protect(8)
DefaultDependencies=no
Before=minio.service
After=local-fs.target

[Service]
Type=simple
ExecStart=/usr/local/bin/minio-protect
Restart=always
RestartSec=5s

# Security hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes

# Requires root for eBPF
User=root
Group=root

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable minio-protect
sudo systemctl start minio-protect
sudo systemctl status minio-protect

Test that protection is working:

# Create a test file
sudo touch /drive1/test-file

# Try to delete as root (should fail)
sudo rm /drive1/test-file
# Expected: rm: cannot remove '/drive1/test-file': Operation not permitted

# Verify file still exists
ls -l /drive1/test-file

Check service status:

# Service status
sudo systemctl status minio-protect

# View logs
sudo journalctl -u minio-protect -f

# Check if MinIO AIStor processes are being tracked
sudo journalctl -u minio-protect | grep "Tracking MinIO process"

The minio-protect daemon automatically:

1. Scans for MinIO AIStor processes every 5 seconds.
2. Identifies processes by binary name containing “minio”.
3. Adds the process ID to the eBPF map of allowed processes.
4. Tracks child processes spawned by MinIO AIStor.
5. Removes stale process IDs when processes exit.

By default, the protection covers directories matching:

• /drive[0-9]+/* (for example, /drive1, /drive2)
• /data[0-9]+/* (for example, /data1, /data2)

The eBPF LSM hooks intercept:

• unlink() - File deletion
• rmdir() - Directory deletion
• rename() - Moving files out of protected directories

If MinIO AIStor fails to start after enabling protection:

# Check SELinux denials
sudo ausearch -m avc -ts recent

# Check MinIO AIStor binary label
ls -Z /usr/local/bin/minio

# Temporarily set permissive mode to diagnose
sudo setenforce 0
# Start MinIO AIStor, then check audit log
sudo ausearch -m avc -ts recent | audit2allow

# Check if minio-protect service is running
sudo systemctl status minio-protect

# Check logs for errors
sudo journalctl -u minio-protect -n 50

# Verify kernel support
/usr/local/bin/minio-protect check

# Restart protection service
sudo systemctl restart minio-protect

If MinIO AIStor cannot write to protected drives:

# Verify directory labels
ls -Z /drive1

# Relabel if needed
restorecon -R -v /drive1

# Check for context mismatches
sudo ausearch -m avc -ts recent | grep minio

# Check if MinIO AIStor PID is tracked
sudo journalctl -u minio-protect | grep "Tracking MinIO"

# Force PID scan
sudo systemctl restart minio-protect

# Wait for MinIO AIStor to be detected
sleep 5

If root can still delete files:

# Verify SELinux is enforcing
getenforce

# Check if policy is loaded
semodule -l | grep minio_storage

# Verify file contexts
ls -Z /drive1

# Verify eBPF program is loaded
sudo bpftool prog list | grep minio

# Verify LSM hooks are attached
sudo bpftool link list

# Check trace log
sudo cat /sys/kernel/debug/tracing/trace_pipe | grep BLOCKED

Storage hardening is one layer of defense. Implement additional measures:

• Regular backups to separate storage
• Filesystem snapshots (ZFS, Btrfs, LVM)
• Restricted SSH access with key-based authentication
• Multi-factor authentication for administrative access
• Monitoring for unauthorized access attempts

Both implementations have known limitations:

semanage fcontext -a -t minio_data_t "/drive9(/.*)?"
restorecon -R -v /drive9

# Remove the policy module
sudo semodule -r minio_storage

# Remove file context rules
sudo semanage fcontext -d "/usr/local/bin/minio"
sudo semanage fcontext -d "/drive1(/.*)?"
# Repeat for other drives

# Restore default labels
sudo restorecon -R -v /usr/local/bin/minio /drive1

# Using setup script
sudo ./setup-ebpf-protection.sh --uninstall

# Or manually:
sudo systemctl stop minio-protect
sudo systemctl disable minio-protect
sudo rm /etc/systemd/system/minio-protect.service
sudo rm /usr/local/bin/minio-protect
sudo systemctl daemon-reload