Shulou(Shulou.com)06/01 Report--

This article shows you how Kafka's message record retention strategy and configuration is, concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Several important configuration items in broker

Log.retention.check.interval.ms

Log.retention.hours

Log.segment.bytes

Log.roll.hours

Message record retention executes basic logic

Kafka messages are stored in the segment file

The operation object of kafka to deal with expired data is segment, which is compressed or deleted, or combined with compression and deletion.

Write data

1.1 the current partition receives the first message to be logged by producer. If there is no segment, kafka will create a new segment.

1.2 check the timestamp of the last insert data of the segment to be written: if the size is greater than the size configured by log.segment.bytes, no more writes are written, a new segment is created, and the write cursor is directed to the newly created segment

1.3 check the size of the segment file to be written: if the time from the creation of the segment to the current time is greater than the time configured by log.roll.hours, it is no longer written, a new segment is created, and the write cursor points to the newly created segment

1.4 write data to the segment file

Check and process expired segment files regularly

2.1 kafka initiates scheduled tasks according to log.retention.check.interval.ms configuration items: regularly check all segment

2.2 check whether segment meets the conditions for performing expiration processing, and if the time between the most recent timestamp of segment and now is greater than the time of log.retention.hours configuration, the segment file will be deleted.

The above is what the message record retention strategy and configuration of Kafka is like. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.