Publish audit logs to Kafka

MinIO AIStor supports publishing audit logs to one or more configured Apache Kafka receivers. MinIO AIStor sends an event for each API operation to the receiver for processing and storage.

The receiver is responsible for correctly processing events, including returning 200OK or similar success messages upon receipt of the event. MinIO AIStor cannot recover events that were successfully sent but not correctly stored on the receiver.

You can configure a new Kafka audit endpoint using either environment variables or runtime configuration settings. If you configure both, MinIO AIStor uses the environment variables.

You can use environment variables to configure one or more Kafka receivers.

The following example code block displays the environment variables commonly used when publishing audit logs to Kafka. Commented variables (starting with a #) are optional. Add these environment variables to the /etc/default/minio environment file on all MinIO AIStor nodes.

MINIO_AUDIT_KAFKA_ENABLE="on"
MINIO_AUDIT_KAFKA_BROKERS="kafka-1.example.net:9092"
MINIO_AUDIT_KAFKA_TOPIC="aistor"
#MINIO_AUDIT_KAFKA_VERSION=""
#MINIO_AUDIT_KAFKA_SASL="on"
#MINIO_AUDIT_KAFKA_SASL_USERNAME="username"
#MINIO_AUDIT_KAFKA_SASL_PASSWORD="password"
#MINIO_AUDIT_KAFKA_SASL_MECHANISM="plain"
#MINIO_AUDIT_KAFKA_SASL_KRB_REALM="myorgrealm"
#MINIO_AUDIT_KAFKA_SASL_KRB_KEYTAB="/etc/minio/keytab"
#MINIO_AUDIT_KAFKA_SASL_KRB_CONFIG_PATH="/etc/minio/krb5.conf"
#MINIO_AUDIT_KAFKA_SASL_KRB_PRINCIPAL="minio"
#MINIO_AUDIT_KAFKA_TLS="on"
#MINIO_AUDIT_KAFKA_TLS_CLIENT_AUTH=0
#MINIO_AUDIT_KAFKA_TLS_SKIP_VERIFY=false
#MINIO_AUDIT_KAFKA_CLIENT_TLS_CERT="/opt/minio/kafka/public.crt"
#MINIO_AUDIT_KAFKA_CLIENT_TLS_KEY="/opt/minio/kafka/private.key"

You can use the mc admin config set command and the audit_kafka configuration key to configure one or more kafka logging targets.

The following example code displays the configuration settings commonly used when publishing audit logs to Kafka. Commented settings (lines starting with a #) are optional.

mc admin config set ALIAS audit_kafka                 \
   brokers="kafka-1.example.net:9092"                 \
   topic="aistor"                                     \
#  version=""                                         \
#  sasl="on"                                          \
#  sasl_username="username"                           \
#  sasl_password="password"                           \
#  sasl_mechanism="plain"                             \
#  sasl_krb_realm="myorgrealm"                        \
#  sasl_krb_keytab="/etc/minio/keytab"                \
#  sasl_krb_config_path="/etc/minio/krb5.conf"        \
#  sasl_krb_principal="minio"                         \
#  tls="on"                                           \
#  tls_client_auth=0                                  \
#  tls_skip_verify=false                              \
#  client_tls_cert="/opt/minio/kafka/public.crt"      \
#  client_tls_key="/opt/minio/kafka/private.key"

For options that require specifying a directory path, ensure the minio-user user and group have read, write, and list access to those resources. Where possible use chown and chmod to limit access and ownership to only the minio-user.

A change made with mc admin config set applies without a server restart. A change made through an environment variable requires a restart.

You can specify multiple Kafka loggers by appending a unique identifier to each group of environment variables or settings. For example, MINIO_AUDIT_KAFKA_BROKERS_PRIMARY or mc admin config set alias audit_kafka:primary.