Shulou(Shulou.com)05/31 Report--

It is believed that many inexperienced people do not know what to do about how to achieve the precise semantic transmission of Kafka. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

We all know that Kafka has a lot of throughput, but will Kafka lose messages? Will you repeat the consumption message?

There are many companies because of business requirements to ensure that messages are not lost and not repeated, such as the real-time monitoring system of drones. When drones break into the airport area, we must immediately report to the police and do not allow messages to be lost. After the drones leave the no-fly zone, we need to lift the alarm in time. If the message is repeated, do we need complex logic to deal with the situation of message duplication ourselves? I am afraid this situation is quite complicated and difficult to deal with. But if we can guarantee the message exactly once, then everything is much easier.

Let's take a brief look at the messaging semantics and the messaging mechanism of kafka.

The first thing we need to understand is message delivery semantic, or messaging semantics.

This is a general concept, that is, the assurance of message delivery in the message delivery process.

There are three categories:

At most once at most: messages may be lost or processed, but only once at most.

May be lost and will not be repeated.

At least once (at least once): messages are not lost, but may be processed multiple times.

Maybe the repetition will not be lost.

Exact delivery once (exactly once): messages are processed and processed only once.

Don't lose it, don't repeat it, just once.

Kafka actually delivers messages twice, once the producer sends a message to kafka, and once the consumer goes to kafka to consume the message.

Both passes will affect the final result.

Both times are accurate, and the final result is accurate.

The message will be lost in one of the two times, or repeated in one time, and the end result may be lost or repeated.

1. Product-side message delivery

This is the code on the producer side:

Properties properties = new Properties ()

Properties.put ("bootstrap.servers", "kafka01:9092,kafka02:9092")

Properties.put ("acks", "all")

Properties.put ("retries", 0)

Properties.put ("batch.size", 16384)

Properties.put ("linger.ms", 1)

Properties.put ("buffer.memory", 33554432)

Properties.put ("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")

Properties.put ("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")

KafkaProducer kafkaProducer = new KafkaProducer (properties)

For (int I = 1; I

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.