The 4.30 GA version (main17386) of the Kafka Snap Pack upgrades the Kafka client libraries from version 2.8.1 to 3.1.1. While the Kafka protocol is designed for forward- and backward-compatibility to allow clients to interoperate with brokers of newer or older versions, recent changes in Kafka to enable idempotent writes by default can result in authorization errors in the Kafka Producer Snap when used with a Kafka cluster whose version is less than 2.8.
If your Kafka cluster is at version 2.8 or greater, this should not be an issue. Otherwise, any one of the following actions will remediate this issue:
-
Upgrade your Kafka cluster to at least version 2.8.
-
Use Kafka’s ACL management to ensure that IdempotentWrite (aka IDEMPOTENT_WRITE) is allowed for the relevant user/principal on the cluster. If this ACL is already present, you should not encounter this issue, even with an older Kafka version.
-
Disable idempotent writes by setting the Kafka Producer configuration property enable.idempotence to false in the Advanced Properties section of the Snap’s account. However, this is not recommended as it would weaken Kafka’s delivery guarantee.
To test for this issue prior to upgrading to the 4.30 GA release, you can use a 4.29 patch version of the Kafka Snap Pack, 429patches17109, which includes the same changes as the 4.30 GA version.
Feel free to reply here or message me directly with any questions.
This content has been added to our Release Notes.
