Kafka Notification Settings

This page documents settings for configuring an Kafka service as a target for Bucket Notifications. See Publish Events to Kafka for a tutorial on using these settings.

You can establish or modify settings by defining:

• an environment variable on the host system prior to starting or restarting the AIStor Server. Refer to your operating system’s documentation for how to define an environment variable.
• a configuration setting using mc admin config set.

If you define both an environment variable and the similar configuration setting, MinIO AIStor uses the environment variable value.

Some settings have only an environment variable or a configuration setting, but not both.

Example

export MINIO_NOTIFY_KAFKA_ENABLE_PRIMARY="on"
export MINIO_NOTIFY_KAFKA_BROKERS_PRIMARY="kafka1.example.net:9092,kafka2.example.net:9092"

export MINIO_NOTIFY_KAFKA_ENABLE_SECONDARY="on"
export MINIO_NOTIFY_KAFKA_BROKERS_SECONDARY="kafka1.example.net:9092,kafka2.example.net:9092"

mc admin config set ALIAS notify_kafka:primary \
   brokers="kafka1.example.net:9092,kafka2.example.net:9092" \
   [ARGUMENT=VALUE ...]

mc admin config set ALIAS notify_kafka:secondary \
   brokers="kafka1.example.net:9092,kafka2.example.net:9092" \
   [ARGUMENT=VALUE ...]

Notice that for configuration settings, the unique identifier appends to notify_kafka only, not to each individual argument.

Settings

Enable

Specify on to enable publishing bucket notifications to a Kafka endpoint. Defaults to off.

Brokers

Specify a comma-separated list of Kafka broker addresses. For example: "kafka1.example.com:2021,kafka2.example.com:2021"

MinIO AIStor checks the health of the specified URL (if it is resolvable and reachable) prior to adding the target. MinIO AIStor adds offline targets without blocking.

The broker list is used only to discover the Kafka cluster. After discovery, MinIO AIStor produces directly to the broker that leads each topic partition, at the address that broker advertises. Removing an unhealthy broker from this list does not stop delivery to it while Kafka still reports it as a partition leader. See Plan for broker failures for details.

Topic

Specify the Kafka topic to which MinIO AIStor publishes events.

SASL

Specify on to enable SASL authentication to the Kafka service endpoint. Defaults to off.

MinIO AIStor ignores the other SASL settings, including the mechanism, the username, and the password, while this setting is off.

SASL username

Specify the SASL username for authenticating to the Kafka service endpoint.

SASL password

Specify the SASL password for authenticating to the Kafka service endpoint.

MinIO AIStor redacts this value when returned as part of mc admin config get.

SASL Kerberos realm

The realm name to use with SASL/GSSAPI (Kerberos) authentication.

SASL Kerberos keytab

The path to the MinIO AIStor keytab to use with SASL/GSSAPI (Kerberos) authentication.

SASL Kerberos config path

The path to the krb5.conf file to use with SASL/GSSAPI (Kerberos) authentication.

SASL Kerberos principal

The principal name without realm suffix to use with SASL/GSSAPI (Kerberos) authentication.

Queue dir

Specify the directory path to enable a persistent event store for undelivered messages, such as /opt/minio/events.

MinIO AIStor stores undelivered events in the specified store while the Kafka server/broker is offline and replays the stored events when connectivity resumes.

Queue limit

Specify the maximum limit for undelivered messages. Defaults to 100000.

The limit applies to each target on each MinIO AIStor server. When the queue is full, MinIO AIStor drops new events for that target and increments the minio_notification_target_failed_events metric. See Size the queue for the outage you want to survive.

Comment

Specify a comment to associate with the Kafka configuration.

SASL mechanism

Specify the SASL mechanism to use for authenticating to the Kafka broker(s). MinIO AIStor supports the following mechanisms:

• SHA512 for SCRAM-SHA-512
• SHA256 for SCRAM-SHA-256
• GSSAPI for Kerberos

Any other value, including an unset one, selects PLAIN.

TLS client auth

Specify the client authentication type of the Kafka broker(s). The following table lists the supported values and their mappings:

See ClientAuthType for more information on each client auth type.

Defaults to 0.

TLS

Specify on to enable TLS connectivity to the Kafka broker(s).

TLS skip verify

Enables or disables TLS verification of the Kafka broker(s) TLS certificates.

• Specify on to disable TLS verification (Default).
• Specify off to enable TLS verification.

Client TLS cert

Specify the path to the client certificate to use for performing mTLS authentication to the Kafka broker(s).

Client TLS key

Specify the path to the client private key to use for performing mTLS authentication to the Kafka broker(s).

Version

Specify the version of the Kafka cluster to assume when performing operations against that cluster. See the sarama reference documentation for more information on this field’s behavior.

Batch size

Specify the integer value to use as the batch size for sending records to Kafka.

Batch commit timeout

The commit timeout for the batch, such as 10s. MinIO AIStor uses this setting only when batch_size is greater than 1.

Defaults to 0s.

Compression codec

Specify the compression codec to use when sending records to Kafka.

Supports the following values:

• none
• snappy
• gzip
• lz4
• zstd

Compression level

Controls the level of compression applied based on the configured compression codec.

Specify an integer value greater than or equal to 0. The effect of the value depends on the selected codec.