Shulou(Shulou.com)06/02 Report--

It is believed that many inexperienced people are at a loss about how to achieve millions of concurrent writes per second in Kafka. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Let's talk about some of the architectural design principles of Kafka, which is also a very frequent technical test site for Internet companies during interviews.

Kafka is a high concurrency and high performance message middleware with high throughput and low latency, which is widely used in the field of big data. A well-configured Kafka cluster can even achieve hundreds of thousands or millions of concurrent writes per second.

So how on earth does Kafka achieve such high throughput and performance? Let's elaborate on this article.

Page caching technology + disk sequential write

First, Kafka writes data to disk every time it receives data, as shown in the following figure:

So here we can not help but have a question, if the data is based on disk storage, frequently write data to the disk file, this performance will be very poor? Everyone must think that disk write performance is extremely poor.

Yes, if it is really as simple as the picture above, then the performance is really poor.

But in fact, Kafka has an excellent and excellent design here, that is, in order to ensure data writing performance, first of all, Kafka is based on the operating system's page cache to achieve file writing.

The operating system itself has a layer of cache called Page Cache, which is a cache in memory. We can also call it OS Cache, which means the cache managed by the operating system itself.

When you write to a disk file, you can write directly to the OS Cache, that is, just write to memory, and then it is up to the operating system to decide when to actually brush the data in the OS Cache into the disk file.

With this step alone, the disk file writing performance can be greatly improved, because it is actually equivalent to writing memory, not writing to the disk. Take a look at the following figure:

And then the other one is when kafka writes data, which is very critical, it is written in disk order.

That is, only append the data to the end of the file, not modify the data at a random location in the file.

The performance of an ordinary mechanical disk is really poor if you write it at random, that is, you can find somewhere in the file to write the data.

But if you append the data sequentially at the end of the file, the performance of this disk sequential write is basically the same as the performance of write memory itself.

So you know, in the figure above, when Kafka writes data, on the one hand, it is based on Page Cache at the OS level to write data, so the performance is very high, in essence, it is writing memory.

On the other hand, it is written sequentially on disk, so the performance is very high even when the data is flushed to disk, which is similar to writing memory.

Based on the above two points, Kafka achieves ultra-high performance of writing data. So if you think about it, if it takes 1 millisecond for Kafka to write a piece of data, is it possible to write 1000 pieces of data per second?

But what if the performance of Kafka is so high that it takes only 0.01ms to write a piece of data? Is it possible to write 100000 pieces of data per second?

Therefore, the core point to ensure that tens of thousands or even hundreds of thousands of data are written per second is to improve the performance of each data write as much as possible, so that more data can be written per unit time and throughput can be improved.

Zero copy technology

When you're done, write this piece, and then talk about consumption.

As you all know, we often consume data from Kafka, so when we consume data, we actually need to read a certain piece of data from the disk file of Kafka and send it to downstream consumers, as shown below:

So if you frequently read data from disk and send it to consumers, where is the performance bottleneck?

Suppose that if Kafka doesn't do any optimization, it simply reads data from disk and sends it to downstream consumers, then the process is as follows:

First check whether the data to be read is in the OS Cache. If not, read the data from the disk file and put it into the OS Cache.

Then copy the data from the OS Cache of the operating system to the cache of the application process, and then copy the data from the cache of the application process to the Socket cache at the operating system level.

* the data is extracted from the Socket cache and sent to the ENI, and * * sent to the downstream consumer.

The whole process, as shown in the following figure:

If you look at the picture above, you can obviously see that there are two unnecessary copies! Once it is copied from the Cache of the operating system into the cache of the application process, and then back to the Socket cache of the operating system from the application cache.

And in order to make these two copies, there are several context switches in between, one is the application is executing, the other is the context switching to the operating system.

So reading data in this way is more performance-consuming. In order to solve this problem, Kafka introduces zero-copy technology when reading data.

In other words, the data in the Cache of the operating system is directly sent to the network card and then transmitted to the downstream consumers. The step of copying the data twice is skipped, and only a descriptor is copied in the Socket cache, not to the Socket cache.

Take a look at the picture below to experience this exquisite process:

With zero copy technology, there is no need to copy the data in OS Cache to the application cache, and then from the application cache to the Socket cache, both copies are omitted, so it is called zero copy.

The Socket cache is simply a descriptor for copying the data, and then the data is sent directly from the OS Cache to the network card, which greatly improves the performance of reading file data during data consumption.

And you will notice that when reading data from disk, you will first look at whether there is any in OS Cache memory, if so, actually read the data directly from memory.

If the Kafka cluster is well tuned, you will find that a large amount of data is written directly to the OS Cache, and then the data is also read from the OS Cache.

It is equivalent to that Kafka provides data writing and reading based entirely on memory, so the overall performance will be extremely high. In fact, the bottom layer of ES is also a large number of high-performance retrieval of massive data based on OS Cache, which is similar to the principle of Kafka.

After reading the above, have you mastered how Kafka can achieve millions of ultra-concurrent writes per second? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.