Publish audit logs to Kafka

AIStor supports publishing audit logs to one or more configured Apache Kafka receivers. AIStor send an event for each API operation to the receiver for processing and storage.

The receiver is responsible for correctly processing events, including returning 200OK or similar success messages upon receipt of the event. AIStor cannot recover events that were successfully sent but not correctly stored on the receiver.

You can configure a new Kafka audit endpoint using either environment variables or runtime configuration settings. If you configure both, AIStor uses the environment variables.

Environment Variable

You can use environment variables to configure one or more Kafka receivers.

The following example code block displays all environment variables related to publishing audit logs to a Kafka. Commented variables (starting with a #) are optional. Add these environment variables to the /etc/default/minio environment file on all AIStor nodes.

MINIO_AUDIT_KAFKA_ENABLE="on"
MINIO_AUDIT_KAFKA_BROKERS="https://kafka-1.example.net"
MINIO_AUDIT_KAFKA_TOPIC="aistor"
#MINIO_AUDIT_KAFKA_VERSION=""
#MINIO_AUDIT_KAFKA_SASL="on"
#MINIO_AUDIT_KAFKA_SASL_USERNAME="username"
#MINIO_AUDIT_KAFKA_SASL_PASSWORD="password"
#MINIO_AUDIT_KAFKA_SASL_MECHANISM="plain"
#MINIO_NOTIFY_KAFKA_SASL_KRB_REALM=myorgrealm
#MINIO_NOTIFY_KAFKA_SASL_KRB_KEYTAB=/etc/minio/keytab
#MINIO_NOTIFY_KAFKA_SASL_KRB_CONFIG_PATH=/etc/minio/krb5.conf
#MINIO_NOTIFY_KAFKA_SASL_KRB_PRINCIPAL=minio
#MINIO_AUDIT_KAFKA_TLS="on"
#MINIO_AUDIT_KAFKA_TLS_CLIENT_AUTH="on"
#MINIO_AUDIT_KAFKA_CLIENT_CERT="/opt/minio/kafka/public.cert"
#MINIO_AUDIT_KAFKA_CLIENT_KEY="/opt/minio/kafka/private.key"
#MINIO_AUDIT_KAFKA_PROXY="https://proxy.example.net"
#MINIO_AUDIT_KAFKA_TLS_SKIP_VERIFICATION=false
#MINIO_AUDIT_KAFKA_HTTP_TIMEOUT=10
#MINIO_AUDIT_KAFKA_HTTP_ENCODING="json"

Configuration Setting

You can use the mc admin config set command and the audit_kafka configuration key to configure one or more kafka logging targets.

The following example code displays all configuration settings related to publishing server logs to a kafka. Commented settings (lines starting with a #) are optional.

mc admin config set ALIAS audit_kafka           \
   brokers="https://kafka-1.example.net"        \
   topic="aistor"                               \
#  version=""                                   \
#  sasl="on"                                    \
#  sasl_username="username"                     \
#  sasl_password="password"                     \
#  sasl_krb_realm="myorgrealm"                  \
#  sasl_krb_keytab=/etc/minio/keytab            \
#  sasl_krb_config_path=/etc/minio/krb5.conf    \
#  sasl_krb_principal="minio"                   \
#  tls="on"                                     \
#  client_cert="/opt/minio/kafka/public.crt"    \
#  client_key="/opt/minio/kafka/private.key"    \
#  proxy="https://proxy.example.net"            \
#  tls_skip_verification=false                  \
#  http_timeout=10                              \
#  http_encoding="json"

For options that require specifying a directory path, ensure the minio-user user and group have read, write, and list access to those resources. Where possible use chown and chmod to limit access and ownership to only the minio-user.

Restart AIStor to apply the new settings.

You can specify multiple kafka loggers by appending a unique identifier to each group of environment variables or settings. For example, MINIO_AUDIT_KAFKA_ENDPOINT_PRIMARY or mc admin config set alias audit_kafka:primary.