Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how to solve the problem that new machines cannot be consumed after new consumer machines are added in spring kafka". In daily operation, I believe many people have doubts about how to solve the problem that new consumer machines in spring kafka can not be consumed all the time. The editor has consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope to answer "how to solve the problem after spring kafka added consumer machines." The confusion of adding machines that can't be consumed all the time helps! Next, please follow the editor to study!

The number of kafka partition online is 24, and the number of consumer cluster machines is also 24, which is normally 1-to-1 consumption.

Due to the backlog of consumption data, it is necessary to expand the capacity of consumer cluster machines. After the new expansion of 24 consumer cluster machines, the new expanded machines cannot be consumed.

After stopping some of the 24 machines that are being consumed online, and adding the corresponding number of newly expanded machines to the load, they are still unable to consume data.

After positioning, it is found that the following code factory.setConcurrency (2) is set to 2, that is, a machine is allowed to open two thread consumption instances at the same time, resulting in the addition of new machines that have been unable to consume.

KafkaListenerContainerFactory kafkaListenerContainerFactory () {logger.info ("batchContainerFactory init"); ConcurrentKafkaListenerContainerFactory factory = new ConcurrentKafkaListenerContainerFactory (); factory.setConsumerFactory (consumerFactory ()); factory.setConcurrency (2); factory.setBatchListener (true); return factory;}

After changing factory.setConcurrency (1) to 1, and after the number of partitions equals the number of consumer machines, the new machines can start consuming data.

At this point, the study on "how to solve the problem that spring kafka has been unable to consume new consumer machines" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.