Ytukyg

I'm trying to make a simple producer-consumer flow over Kafka, using node-rdkafka

I'm using debug: 'all' mode, this is what I get from the logs:

Producer: test [0]: MessageSet with 1 message(s) delivered

Consumer: Fetch topic test [0] at offset 38 (v2)

The fact that the consumer is changing offset when message is produced making me believe the connection with the broker is setup and authenticated correctly.

However, for some reason I don't get the message itself in the consumer.

This event is never called:

consumer.on('data', function(m) {

    console.log("consumed", m)

});

I created a demo project for testing, you need to have a Kafka broker that supports SASL_SSL protocol in order to use it:

https://github.com/guysegal/kafka-example

Specifically, this is the consumer code:

https://github.com/guysegal/kafka-example/blob/master/src/consumer.ts

and the producer code:

https://github.com/guysegal/kafka-example/blob/master/src/producer.ts

You may want to set the value 'auto.offset.reset': 'earliest' and start the consumer again.

auto.offset.reset 

This property decides the consumer starting position in case consumer starting with group which does not have last known state. For e.g. in case of new group-id id.

The reason for changing the value of auto.offset.reset because if you produce first and starts consumer after that then topic offset has already increased and consumer will start with latest offset(increased offset) and reads the message from that point onward.

While in case of 'earliest' consumer starts from first available message on topic.