Storage Hardening

MinIO AIStor supports kernel-level storage protection to prevent accidental deletion of data directories by system administrators while allowing the MinIO AIStor process full control over managed drives. This protection is critical for production environments where a mistyped rm -rf /drive* command could result in catastrophic data loss.

Storage hardening provides:

• MinIO AIStor process: Full read/write/delete access to managed drives.
• Host OS (including root): Cannot delete protected data. With SELinux, other processes have read-only access to protected directories. With eBPF LSM, other processes are blocked from deleting or renaming files owned by a protected UID.
• Kernel-level enforcement: Protection cannot be bypassed through normal filesystem operations.
• No performance impact: Both implementations add negligible overhead.

MinIO AIStor provides two storage protection implementations:

Recommendations:

• RHEL/CentOS/Fedora: Use SELinux (better integration, native support)
• Ubuntu/Debian: Use eBPF LSM (default kernel support, no SELinux required)
• Other distributions: Use eBPF LSM for universal compatibility

Use SELinux-based storage protection on RHEL, CentOS, Fedora, and other SELinux-enabled systems.

• RHEL 8/9, CentOS 8/9, Fedora, or any Linux distribution with SELinux.
• SELinux in enforcing or permissive mode (not disabled).
• SELinux policy development tools.

Check SELinux status:

getenforce

The output should be Enforcing or Permissive. If the output is Disabled, enable SELinux first:

# Edit SELinux config
sudo vi /etc/selinux/config

# Set SELINUX=enforcing
SELINUX=enforcing

# Reboot required
sudo reboot

Install required tools:

sudo dnf install -y policycoreutils-python-utils selinux-policy-devel

Follow these steps to create and install the SELinux policy.

Create a file named minio_storage.te:

policy_module(minio_storage, 1.0.0)

require {
    type unconfined_t;
    type user_t;
    type sysadm_t;
    type init_t;
    class dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
    class file { append create getattr ioctl link lock map open read rename setattr unlink write };
}

# Define MinIO domain
type minio_t;
type minio_exec_t;
type minio_data_t;

# MinIO binary type
files_type(minio_exec_t)

# MinIO data directories type
files_type(minio_data_t)

# Allow MinIO domain to be entered from init
init_daemon_domain(minio_t, minio_exec_t)
domain_type(minio_t)

# MinIO process has FULL access to minio_data_t
allow minio_t minio_data_t:dir { add_name create getattr ioctl link lock open read remove_name rename reparent rmdir search setattr unlink watch watch_reads write };
allow minio_t minio_data_t:file { append create getattr ioctl link lock map open read rename setattr unlink write };

# All other processes can ONLY read, CANNOT delete or write
allow unconfined_t minio_data_t:dir { getattr open read search };
allow unconfined_t minio_data_t:file { getattr open read };
allow user_t minio_data_t:dir { getattr open read search };
allow user_t minio_data_t:file { getattr open read };
allow sysadm_t minio_data_t:dir { getattr open read search };
allow sysadm_t minio_data_t:file { getattr open read };

# MinIO network access
corenet_tcp_bind_generic_node(minio_t)
corenet_tcp_connect_http_port(minio_t)
corenet_tcp_bind_http_port(minio_t)
corenet_tcp_bind_generic_port(minio_t)
corenet_udp_bind_generic_port(minio_t)

# MinIO system permissions
kernel_read_system_state(minio_t)
dev_read_urand(minio_t)
fs_getattr_xattr_fs(minio_t)
files_read_etc_files(minio_t)
miscfiles_read_localization(minio_t)
sysnet_dns_name_resolve(minio_t)

# Allow MinIO to execute itself
allow minio_t minio_exec_t:file { execute execute_no_trans getattr map open read };

# Compile the policy module
checkmodule -M -m -o minio_storage.mod minio_storage.te

# Package the module
semodule_package -o minio_storage.pp -m minio_storage.mod

# Install the policy module
semodule -i minio_storage.pp

# Verify installation
semodule -l | grep minio_storage

# Set file context for MinIO AIStor binary
semanage fcontext -a -t minio_exec_t "/usr/local/bin/minio"

# Apply the label
restorecon -v /usr/local/bin/minio

# Verify the label
ls -Z /usr/local/bin/minio

The output should show system_u:object_r:minio_exec_t:s0.

For each drive that MinIO AIStor manages:

# Add file context rules for each drive
semanage fcontext -a -t minio_data_t "/drive1(/.*)?"
semanage fcontext -a -t minio_data_t "/drive2(/.*)?"
semanage fcontext -a -t minio_data_t "/drive3(/.*)?"
semanage fcontext -a -t minio_data_t "/drive4(/.*)?"

# Apply labels recursively
restorecon -R -v /drive1 /drive2 /drive3 /drive4

# Verify labels
ls -Z /drive1

The output should show system_u:object_r:minio_data_t:s0.

Test that protection is working:

# Create a test file (as MinIO AIStor or with proper context)
sudo -u minio-user touch /drive1/test-file

# Try to delete as root (should fail)
sudo rm /drive1/test-file
# Expected: rm: cannot remove '/drive1/...': Permission denied

Check that processes are running in the correct SELinux domain:

# Check MinIO AIStor process context
ps -eZ | grep minio
# Expected: system_u:system_r:minio_t:s0

# Check file contexts
ls -Z /drive1
# Expected: system_u:object_r:minio_data_t:s0

Use eBPF LSM-based storage protection on Ubuntu, Debian, and other distributions without SELinux.

• AIStor-compatible Linux kernel with eBPF LSM support.
• CONFIG_BPF_LSM=y kernel configuration.
• lsm=...,bpf kernel boot parameter.
• Root access for installation.

Check kernel support:

# Check kernel version
uname -r

# Check if eBPF LSM is enabled
cat /sys/kernel/security/lsm | grep -q bpf && echo "Supported" || echo "Not supported"

# Check kernel configuration (if available)
grep CONFIG_BPF_LSM /boot/config-$(uname -r)

If eBPF LSM is not in the LSM list, enable it via kernel boot parameters.

# Edit GRUB configuration
sudo vi /etc/default/grub

# Add or modify GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash lsm=lockdown,yama,integrity,apparmor,bpf"

# Update GRUB
sudo update-grub

# Reboot
sudo reboot

# Edit GRUB configuration
sudo vi /etc/default/grub

# Add or modify GRUB_CMDLINE_LINUX
GRUB_CMDLINE_LINUX="... lsm=selinux,bpf"

# Update GRUB
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# Reboot
sudo reboot

sudo apt-get update
sudo apt-get install -y \
    clang \
    llvm \
    libbpf-dev \
    linux-tools-generic \
    linux-headers-$(uname -r) \
    golang-go

sudo dnf install -y \
    clang \
    llvm \
    libbpf-devel \
    bpftool \
    kernel-devel \
    golang

The minio-lsm-bpf daemon uses a pure UID-based protection model. You pass it one or more UIDs to protect using the -u option, typically the UID of the user that runs MinIO AIStor. The daemon then protects all files owned by those UIDs, regardless of their location on disk:

1. The daemon attaches eBPF programs to the kernel LSM hooks.
2. When a process attempts to delete a file, the hook checks whether the file is owned by a protected UID.
3. If the file is owned by a protected UID, the operation is allowed only when the deleting process runs as the same UID.
4. All other processes, including root, are blocked from deleting files owned by a protected UID.

The eBPF LSM hooks intercept:

• unlink() - File deletion.
• rmdir() - Directory deletion.
• rename() - Moving or renaming files.

Because protection is based on file ownership rather than path, any drive whose files are owned by a protected UID is covered automatically.

Follow these steps to build, install, and run the protection daemon from the scripts/ebpf directory of the MinIO AIStor source tree.

# Check eBPF LSM support
if ! cat /sys/kernel/security/lsm | grep -q bpf; then
    echo "eBPF LSM not enabled - add 'bpf' to kernel boot parameters"
    exit 1
fi

echo "Kernel support: OK"

You can also run make check-kernel from the scripts/ebpf directory to verify that the system is ready.

cd scripts/ebpf

# Verify the required build tools are present
make check-tools

# Build the binary (default 'all' target)
make

# Verify the binary was created
ls -lh minio-lsm-bpf

# Install to system path
sudo install -m 0755 minio-lsm-bpf /usr/local/bin/

Determine the UID that MinIO AIStor runs as:

# Get the UID of minio-user (for example, 1000)
id -u minio-user

Create /etc/systemd/system/minio-lsm-bpf.service, replacing 1000 with the UID from the previous command:

[Unit]
Description=MinIO eBPF LSM Protection
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/minio-lsm-bpf -u 1000
Restart=always
RestartSec=10
LimitMEMLOCK=infinity
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

To protect more than one UID, repeat the -u option, for example ExecStart=/usr/local/bin/minio-lsm-bpf -u 1000 -u 1001.

The daemon supports the following options:

• -u UID - UID whose files are protected (required, can be repeated).
• -v - Enable verbose logging (logs allowed operations in addition to blocked operations).

sudo systemctl daemon-reload
sudo systemctl enable --now minio-lsm-bpf
sudo systemctl status minio-lsm-bpf

Test that protection is working:

# Create a test file owned by the protected user
sudo -u minio-user touch /drive1/test-file

# Try to delete as root (should fail)
sudo rm /drive1/test-file
# Expected: rm: cannot remove '/drive1/test-file': Operation not permitted

# Verify the file still exists
ls -l /drive1/test-file

# Delete as the file owner (should succeed)
sudo -u minio-user rm /drive1/test-file

Check service status:

# Service status
sudo systemctl status minio-lsm-bpf

# View logs (blocked operations appear in real time)
sudo journalctl -u minio-lsm-bpf -f
# Example output:
# BLOCKED unlink: op_pid=1234 op_uid=0 comm=rm owner_uid=1000

If MinIO AIStor fails to start after enabling protection:

# Check SELinux denials
sudo ausearch -m avc -ts recent

# Check MinIO AIStor binary label
ls -Z /usr/local/bin/minio

# Temporarily set permissive mode to diagnose
sudo setenforce 0
# Start MinIO AIStor, then check audit log
sudo ausearch -m avc -ts recent | audit2allow

# Check if the minio-lsm-bpf service is running
sudo systemctl status minio-lsm-bpf

# Check logs for errors
sudo journalctl -u minio-lsm-bpf -n 50

# Verify kernel support
cat /sys/kernel/security/lsm | grep -q bpf && echo "Supported" || echo "Not supported"

# Restart protection service
sudo systemctl restart minio-lsm-bpf

If MinIO AIStor cannot write to protected drives:

# Verify directory labels
ls -Z /drive1

# Relabel if needed
restorecon -R -v /drive1

# Check for context mismatches
sudo ausearch -m avc -ts recent | grep minio

# Confirm the protected UID matches the MinIO AIStor user
id -u minio-user
sudo systemctl show minio-lsm-bpf -p ExecStart

# Confirm drive ownership
ls -ln /drive1

# Check logs for blocked operations
sudo journalctl -u minio-lsm-bpf -n 50

If root can still delete files:

# Verify SELinux is enforcing
getenforce

# Check if policy is loaded
semodule -l | grep minio_storage

# Verify file contexts
ls -Z /drive1

# Verify eBPF program is loaded
sudo bpftool prog list | grep minio

# Verify LSM hooks are attached
sudo bpftool link list

# Check the service logs for blocked operations
sudo journalctl -u minio-lsm-bpf | grep BLOCKED

Storage hardening is one layer of defense. Implement additional measures:

• Regular backups to separate storage
• Filesystem snapshots (ZFS, Btrfs, LVM)
• Restricted SSH access with key-based authentication
• Multi-factor authentication for administrative access
• Monitoring for unauthorized access attempts

Both implementations have known limitations:

semanage fcontext -a -t minio_data_t "/drive9(/.*)?"
restorecon -R -v /drive9

sudo systemctl daemon-reload
sudo systemctl restart minio-lsm-bpf

# Remove the policy module
sudo semodule -r minio_storage

# Remove file context rules
sudo semanage fcontext -d "/usr/local/bin/minio"
sudo semanage fcontext -d "/drive1(/.*)?"
# Repeat for other drives

# Restore default labels
sudo restorecon -R -v /usr/local/bin/minio /drive1

sudo systemctl disable --now minio-lsm-bpf
sudo rm /etc/systemd/system/minio-lsm-bpf.service
sudo systemctl daemon-reload
sudo rm /usr/local/bin/minio-lsm-bpf