Shulou(Shulou.com)06/03 Report--

This article mainly introduces "what is the storage method of Kafka". In the daily operation, I believe that many people have doubts about the storage method of Kafka. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "what is the storage method of Kafka?" Next, please follow the editor to study!

Kafka relies on the file system (or, more fundamentally, the disk) to store and cache messages. In our impression, the speed perception of each storage medium is roughly the same as shown in the following figure, and the higher the level, the faster the speed. Obviously, the disk is in an awkward position, which makes us wonder whether Kafka can provide competitive performance in this form of persistence. In the traditional message middleware RabbitMQ, memory is used as the default storage medium, while disk is used as the alternative medium to achieve the characteristics of high throughput and low latency. However, the fact that a disk can be faster or slower than we expect depends entirely on how we use it.

The test results show that the linear (sequential) write speed of a disk cluster composed of 6 7200r/min RAID-5 arrays can reach 600MB/s, while the random write speed is only 100KB/s, the performance difference between the two is 6000 times. The operating system can make deep optimizations for linear reads and writes, such as read-ahead (reading a large disk block into memory in advance) and post-write (write-behind, combining many small logical writes to form a large physical write operation). The speed of sequential disk writing is not only faster than random disk writing, but also faster than random memory writing, as shown in the following figure.

The charm of page caching

Kafka uses file append to write messages, that is, new messages can only be appended at the end of the log file, and written messages are not allowed to be modified. This is a typical sequential disk write operation, so even if Kafka uses disk as storage medium, it can carry throughput that should not be underestimated. But this is not the only factor that makes Kafka competitive enough in terms of performance, and we might as well continue to analyze it.

Page cache is a major disk cache implemented by the operating system, which is used to reduce the operation of disk Imax O. Specifically, it is to cache the data in the disk into memory and change the access to the disk into the access to memory. In order to make up for the difference in performance, modern operating systems are increasingly "aggressive" in using memory as a disk cache, and will even be more than happy to use all available memory as a disk cache, so that there is almost no performance loss when memory is reclaimed. All reads and writes to the disk will also go through a unified cache. Phellodendron mandshurica (Thunb.)

When a process is ready to read the contents of a file on disk, the operating system will first check whether the page (page) of the read data is in the page cache (pagecache), and if it exists (hit), it will return the data directly, thus avoiding the Icano operation on the physical disk; if not, the operating system will issue a read request to the disk and store the read data page in the page cache, and then return the data to the process.

Similarly, if a process needs to write data to disk, the operating system will also detect whether the page corresponding to the data is in the page cache, and if it does not exist, it will first add the corresponding page to the page cache, and finally write the data to the corresponding page. The modified page becomes a dirty page, and the operating system will write the data in the dirty page to disk at an appropriate time to maintain data consistency.

The vm.dirty_background_ratio parameter in the Linux operating system is used to specify that when the number of dirty pages reaches how many percent of the system memory, the background write-back process such as pdflush/flush/kdmflush will be triggered to deal with dirty pages. It is generally set to a value of less than 10, but it is not recommended to set it to 0. There is also a vm.dirty_ratio parameter corresponding to this parameter, which is used to specify that dirty pages have to be processed when the number of dirty pages reaches how many percent of the system memory, and in the process, the new Imax O request will be blocked until all dirty pages are washed to disk. Readers who are interested in dirty pages can also check the instructions for the use of parameters such as vm.dirty_expire_centisecs, vm.dirty_writeback.centisecs and so on. Phellodendron mandshurica (Thunb.)

For a process, it caches the data needed for processing within the process, but the data may still be cached in the operating system's page cache, so the same data may be cached twice. And it is difficult to disable page caching unless you use the Direct Iamp O method. In addition, people who have used Java generally know two facts: the memory overhead of objects is very high, usually several times or more than the real data size, and space utilization is low; garbage collection of Java becomes slower and slower as the data in the heap increases. Based on these factors, using the file system and relying on the page cache is obviously better than maintaining an in-process cache or other structure, at least we can save a copy of the internal cache consumption. at the same time, more space can be saved by using compact bytecodes instead of using objects. In this way, we can use 28GB to 30GB memory on 32GB machines without worrying about the performance problems caused by GC.

In addition, the page cache remains valid even if the Kafka service is restarted, but the in-process cache needs to be rebuilt. This also greatly simplifies code logic, because maintaining consistency between page caches and files is left to the operating system, which is more secure and efficient than in-process maintenance.

Page caching is widely used in Kafka, which is one of the important factors for Kafka to achieve high throughput. Although messages are first written to the page cache, and then the operating system is responsible for specific flushing tasks, synchronous flushing and intermittent forced flushing (fsync) are also provided in Kafka, which can be controlled by parameters such as log.flush.interval.messages, log.flush.interval.ms and so on.

Synchronous flushing can improve the reliability of messages and prevent the loss of messages that are in the page cache and are not written to disk in time due to anomalies such as machine power failure. However, the author does not recommend this. The task of disk brushing should be assigned to the operating system, and the reliability of messages should be guaranteed by the multi-copy mechanism, rather than by synchronous flushing, which seriously affects performance.

The Linux system uses a portion of the disk as the swap partition so that processes can be scheduled: the currently inactive processes are called into the swap partition to free up memory for active processes. For Kafka, which uses a lot of system page cache, this kind of memory swap should be avoided as far as possible, otherwise it will have a great negative impact on its performance in all aspects.

We can adjust it by modifying the vm.swappiness parameter (Linux system parameter). The upper limit of the vm.swappiness parameter is 100. it means to actively use the swap partition and move the data in memory to the swap partition in a timely manner. The lower bound of the vm.swappiness parameter is 0, which means that swapping should not occur under any circumstances (the meaning of vm.swappiness = 0 is different in different versions of the Linux kernel. This is the latest explanation after the change). In this way, some processes will be abruptly aborted according to certain rules when memory is exhausted. The author recommends that the value of this parameter be set to 1, which preserves the mechanism of swap while maximizing its impact on Kafka performance.

At this point, the study on "what is the storage method of Kafka" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.