Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what are the Kafka log cleaning strategies", which is easy to understand and well organized. I hope it can help you solve your doubts. Let me lead you to study and learn this article "what are the Kafka log cleaning strategies?"

Currently, there are two main strategies for Kafka log cleanup:

Log deletion

Log compression: log compression is to retain the last message based on key.

Kafka provides the log.cleanup.policy parameter to configure accordingly. The default value is delete, and you can also select compact.

Log deletion configuration default value description log.retention.check.interval.ms300000 (5 minutes) detection frequency log.retention.hours168 (7 days) log retention time hours log.retention.minutes

Log retention time minutes log.retention.ms

Log retention time millisecond file.delete.delay.ms60000 (1 minute) delay deletion time log.retention.bytes-1 infinite run retention log file maximum log.retention.bytes1073741824 (1G) log file maximum

Kafka periodically detects whether logs need to be deleted according to the detection frequency. Log deletion policies are mainly concentrated as follows:

Time-based deletion policy

Delete policy based on file size

Based on log file start offset

Delete based on time

Kafka looks for the last record in the timestamp index file of the log segment, and if the last timestamp is less than 0, the last modification time is taken.

After confirming the log segments that need to be deleted, you need to delete the following:

Remove the log segments to be deleted from the ConcurrentSkipListMap of the log segments maintained in the log object to ensure that there are no threads to read these log segments.

Add a .delete suffix to all files in the log segment

There will be a deferred task named "delete-file" in Kafka to delete these invalid log data

If there is also data in the current log segment that needs to be deleted, kafka will first segment it, create a new active log segment, and then perform the delete operation.

Delete based on size

Calculate the sum of log sizes that need to be deleted (current log size-maximum allowed log files)

Then start with the first log segment to find a collection of log segments that can be deleted

Finally, delete operation is performed.

Based on log file start offset

By default, the initial offset of the log file is equal to the base offset of the first log segment, but this value is constantly changing as the data is deleted.

The judgment rule based on the initial offset of the log file is that if the starting offset of the next log segment of a log segment is less than the initial offset of the log file, then the log segment can be added to the deletion queue and the deletion can be performed finally.

The above is all the contents of the article "what are the Kafka log cleaning strategies?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.