Shulou(Shulou.com)06/02 Report--

Kafka how to ensure that the message is not lost and repeated, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Repeat consumption on the consumer side:

Establish a deduplicated table

Data loss on the consumer side:

Turn off the auto-commit offset and be shifted after processing

Repeat transmission on the production side:

This is not important. Consumers can judge the weight from the deduplicated table before consumption.

Data loss on the production side:

1. The asynchronous buffer is full, so it is blocked there, waiting for the buffer to be available, and the buffer cannot be emptied

2. After sending the message, call back the function. Send the next message after it is sent successfully.

Send failure is recorded in the log, waiting for the timing script (timer) to scan.

(failure to send may not really fail, but no feedback has been received, and scheduled scripts may be resent)

Resolution strategy:

How to ensure order (synchronous transmission can ensure order):

Synchronous sending mode: after sending a message, you must block and wait for the notification before sending the next message.

Asynchronous send mode: write all the way to the buffer, and then write one to the queue

Both have their own advantages and disadvantages:

Although the throughput of the synchronous sending mode is small, it can not only ensure that the message will not be lost, but also ensure the order by sending one message after receiving the acknowledgement.

Reasons for kafka data loss:

There are three states (0 (no need to wait for confirmation), 1 (leader confirmation),-1 (all confirmation) to guarantee that the message is safe in production

If the configuration status is 1 (only if the leader is successfully written), if the leader partition fails, the data will be lost.

When using asynchronous mode, when the cache area is full

If the configuration status is 0 (as soon as the buffer pool is full, the messages in the buffer pool will be emptied), the data will be discarded immediately.

If the configuration status is-1

Because ISR may shrink to contain only one Replica, setting the parameter to all does not necessarily prevent data loss

Ways to avoid data loss during data production:

As long as the above two situations can be avoided, then the message can be guaranteed not to be lost.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.