As Mentioned in my previous articles we are working on an application which processes more than 20000 requests per minute. We are using Kafka heavily in our application for different reasons especially for implementing Back-Pressure. I am able to find several articles for writing consumer, producer in Node.js using kafka-node client.
One consumer is not enough to process all requests. We need to use multiple consumers so we need to create multiple partitions too. We started getting problems with the simple consumer because with simple consumer we will be constrained to use one partition per consumer. It doesn’t make any sense to create the number of partitions based on number of consumers we want to run. Then we don’t have room for auto-scaling.
With simple producer also we started getting issues. One of our microservice consumes the messages from Kafka and transform them into the desired
format and send them back to Kafka in a different topic which will be consumed by our analytics application(Druid). This transformation doesn’t take much time and with the simple producer, we are unnecessarily sending a request for each message to Kafka. Here Batch producer make sense.
We didn’t find any good article for writing ConsumerGroup and Batch Producer code. Thanks to kafka-node test cases, We were finally able to write code by reading them and understanding them. I want to share the code and my experience around this. Without wasting much time lets get started
I am not going to explain basic things here. I am assuming you know already about Kafka, Node.js, Consumer, and Producer and you run into problems just like me.
Code for ConsumerGroup Initiation
You can use this code for consumer group initiation. The important thing here is “protocol”. This defines how partitions allocated to this consumer group. For each partition, one consumer group is assigned at Kafka end. One consumer group can read multiple partitions but not the other way around. Rebalancing and assignment of partitions to consumer group part taken care by Kafka in “roundrobin” case.
Code for Producer & Batch Producer
Some places we need Producer and some places we need Batch Producer. That’s why here wrote code to support both. The most important thing here is 3 properties/arguments.
{requireAcks: 0} — For Batch Producer, you need to make this sacrifice. You won’t get acknowledgment.
{noAckBatchSize: 5000000, //5 MB, noAckBatchAge: 5000 // 5 Sec} — There 2 are conditions for sending messages to Kafka from Batch Producer. These 2 properties are actually batch.size, linger.ms. If the total messages size at producer side reach 5 MB or 5 sec wait over then Batch producer automatically
sends these messages to Kafka.
Peace. Happy Coding.