I am trying to consume messages from a kafka topic on Amazon MSK (Kafka Version 2.2.1) using NiFi ConsumeKafka_2_6 processor (NiFi version 1.12.1).In ConsumeKafka processor I am getting the warning as "org.apache.kafka.clients.NetworkClient [Consumer clientId=console-consumer-34567-5, groupId=nifi-consumer] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected".What can be the cause of this issue?Is this due to version mismatch of kafka client used or due to connectivity issues.

3 Answers

It's a connectivity problem or you configured the Nifi processor incorrectly

You're not connecting to MSK, you're trying to connect Nifi to itself

Bootstrap broker localhost:9092

Also, yes, you might have some issues using the 2_6 consumer with a 2.2 cluster, but very minor as the core functions should still work

4

From aws cli we use the JVM truststore to talk to the MSK cluster. So from NiFi application also I was supposed to use that. I have configured the security protocol as SSL and configured JVM truststore in SSLContextService. Then I was able to consume messages from Kafka.

There can be multiple reason leading to this error. In my case, AWS MSK IAM auth enabled, I was missing the client.properties file while running kafka-console-consumer.sh script --consumer.config path/to/client.properties

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.