Shulou(Shulou.com)06/02 Report--

This article mainly introduces "why Kafka is so fast". In daily operation, I believe many people have doubts about why Kafka is so fast. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubt of "why Kafka is so fast"! Next, please follow the editor to study!

Why is Kafka so fast?

Fast is a relative concept, there is no harm without comparison, so usually we say that Kafka is a message queue that occurs IO relative to our common ActiveMQ,RabbitMQ, and mainly relies on IO for information transmission.

Message queues like ZeroMQ, which basically rely solely on memory for information flow delivery, are certainly faster, but such message queues are only used in special scenarios and are not included in the comparison.

So when we say Kakfa is fast, it is usually based on the following scenarios:

Throughput: when we need to process hundreds of thousands of millions of Message per second, it is faster than other MQ,Kafka.

High concurrency: when there are millions and tens of millions of Consumer, Kafka will have more Producer and Consumer under the same configuration.

Disk lock: compared with other MQ,Kafka when performing IO operations, there are fewer scenarios in which IO is synchronously locked and the waiting time is shorter.

So based on the above points, let's discuss in detail why Kafka is coming soon.

Push-pull Model of message queue

First of all, if we simply look at "Kafka fast" from the perspective of Consumer, it is a pseudo-proposition, because compared with other MQ,Kafka to generate a Message from Producer to Consumer to consume this Message, its time must be greater than or equal to other MQ.

The reasons behind this involve two models of message queue design:

Push model

Pull model

As shown in the following figure:

For pull models, after Producer generates Message, it will actively send it to MQ Server. In order to improve performance and reduce expenses, some Client will also be designed to be sent in batches.

However, regardless of whether it is a single message or a batch, Producer will actively push messages to MQ Server.

When the MQ Server receives the message, MQ Server will not actively send the message to the Consumer for the pull model, and the Offset,Consumer, which does not maintain and record the message, will automatically set a timer to the server to ask if a new message has been generated.

Usually, the time is no more than once asked by 100ms. Once a new message is generated, it will be synchronized locally, and the Offset will be modified and recorded. The server can assist in storing the Offset, but will not actively record and verify the rationality of the Offset.

At the same time, Consumer can maintain offset completely independently in order to achieve custom information reading.

For the push model, after receiving the Message, the server will first record the information of the message and query the Consumer of the corresponding message from its own meta-information database.

Because the server and Consumer establish a long link when linking, you can send messages directly to Consumer.

Kafka is a message queue based on pull model, so from the point of view of getting messages from Consumer, the delay will be less than or equal to the polling period, so it will have higher message acquisition delay than the message queue of push model, but the push model also has its problems.

First of all, because the server needs to record the meta-information of the corresponding Consumer, including who the message should be sent and how much the Offset is, and the need to push the message to the Consumer, it will inevitably bring a series of problems.

What if the network is not good at this moment, the Consumer is not received, and the message is not sent successfully? Assuming the message has been sent, how do I know if it has been received?

Therefore, the server and Consumer need to first multi-layer confirmation password, in order to achieve at least one consumption, only one consumption and so on.

Pull models such as Kafka leave this function to Consumer to maintain automatically, so the server reduces unnecessary expenses.

So from the perspective of equal resources, Kafka will have more Producer and Consumer with links, greatly reducing message congestion, so it looks faster.

OS Page Cache and Buffer Cache

There is nothing new in the sun, and for a framework, there are usually only a few tricks that can be used to run faster, and Kafka is taking this to the extreme.

One of them is Cache that maximizes the use of OS, mainly Page Cache and Buffer Cache.

Students who use Linux are usually no stranger to these two Cache. For example, when we execute the free command under Linux, we will see the following output:

The picture is from the Internet.

There will be two columns named buffers and cached, and a row named "- / + buffers/cache", which are explained as follows:

Pagecache: file system-level cache, where the content read from the disk is stored here, so that the program reads the contents of the disk very quickly. For example, when you use commands such as grep and find of Linux to find content and files, the first time will be much slower, and the second execution will be many times faster, almost instantly.

In addition, after the data of page cache is modified, that is, dirty data, when the time comes to write to disk, it will be transferred to buffer cache instead of writing directly to disk.

The value we see in the cached column represents the current page cache (page cache) usage, the page data of the page cache file, the page is a logical concept, so page cache is at the same level as the file system.

Buffer cache: the buffer of a block device such as a disk. This part of the memory is written to the disk.

The buffers column represents the current block cache (buffer cache) usage, and buffer cache is used to cache block data for block devices, such as disks. Block is a physical concept, so buffer cache is at the same level as the block device driver.

Both of them are used to speed up the data IO, mark the written page as dirty, and then store flush externally. When reading the data, read the cache first, and then go to the external storage to read if it is missed, and add the read data to the cache.

The operating system always actively uses all free memory for Page Cache and Buffer Cache, and when OS runs out of memory, it will also use algorithms such as LRU to eliminate cache pages.

With the above concepts, we can see how Kafka takes advantage of this feature.

First, for a data IO, the following process usually occurs:

The operating system copies data from disk to Page Cache in the kernel area.

The user program copies the Page Cache of the kernel area to the user area cache.

The user program copies the cache of the user area to the Socket cache.

The operating system copies the data in the Socket cache to the Buffer of the network card and sends the data.

It can be found that an IO request operation takes 2 context switches and 4 system calls, while the same data is copied multiple times in the cache, in fact, the copy can be carried out directly in the kernel state.

That is, omit the second and third steps, and become like this:

Because of the process of modifying data in this way, Kafka refers to this process at the beginning of its design, making use of the Page Cache of OS to copy the data as much as possible to minimize the operation on disk.

If the Kafka production and consumption match well, then the data is completely in memory, which will greatly improve the throughput of the cluster.

In the early operating system, Page Cache and Buffer Cache were two separate pieces of Cache, but later found that the same data could be Cache twice, so in most cases the two were merged into one.

Although Kafka is written in the JVM language and cannot run without the GC of JVM and JVM, Kafka does not manage the cache itself, but directly uses OS's Page Cache as the cache.

This brings the following benefits:

Everything in JVM is an object, so regardless of the size of the object, there is always some extra JVM object metadata wasting space.

JVM's own GC is not manually controlled by the program, so if you use JVM as a cache, it will reduce the throughput of the entire system when you encounter large objects or frequent GC.

If the program exits or restarts abnormally, all caches will be invalidated, which will affect the fast recovery under the disaster recovery framework. Because Page Cache is the Cache of OS, even if the program exits, the cache still exists.

So Kafka optimizes the IO process to make full use of Page Cache, which consumes less time, has higher throughput, and is faster than other MQ.

A picture is used to describe the relationship between the three as follows:

When there is little difference between the rates of Producer and Consumer, Kafka can almost complete the transmission of information without dropping the disk.

Append sequential write

In addition to the previous important features, Kafka also has a design, that is, the sequence of append writes to the persistent storage of data. Kafka only appends messages sequentially when dropping messages to the Partition files of each Topic, making full use of the disk sequential access fast feature.

The picture is from the Internet.

The file storage of Kafka is stored according to the Partition under Topic. Each Partition has its own sequence file, and the sequence of each Partition is not shared. The main division is Hash according to the Key of the message to determine which partition to fall on.

Let's first explain the nouns of Kafka in detail in order to fully understand its characteristics:

The server used to process messages in Broker:Kafka is also a node of the Kafka cluster, and multiple nodes form a Kafka cluster.

Topic: a message topic. Each business system or Consumer needs to subscribe to one or more topics to get the message. Producer needs to make it clear that the Topic to which the message occurs is equal to the password name of the message transmission.

Partition: a Topic will be split into multiple Partition and landed on disk, and files will be stored according to the folder created by the corresponding partition ID under the storage directory configured by Kafka, the largest storage unit visible to the disk.

Segment: a Partition will have multiple Segment files to actually store content.

Offset: each Partition has its own independent sequence number, and its scope is only under the current Partition, which is used to read the contents of the corresponding file.

Leader: each Topic needs a Leader to be responsible for the writing of the Topic information and the maintenance of data consistency.

Controller: each Kafka cluster will choose a Broker to act as Controller, decide who the Leader of each Topic is, monitor the changes of cluster Broker information, and maintain the health of the cluster.

You can see that the final landing to the disk are Segment files, each Partion (directory) is equivalent to a giant file is evenly distributed to multiple equal-size Segment (segment) data files.

However, the number of segment file messages per segment is not necessarily equal, which makes it convenient for the old segment file to be deleted quickly.

Because Kafka processes messages to Partition, you only need to maintain the order corresponding to Partition, and Segment can maintain its state separately.

The file of Segment consists of index file and data file. The suffixes of landing on disk are .index and .log, and the files are generated by serial number, as shown below:

The picture is from the Internet.

Index maintains the physical address of the data, while data stores the offset address of the data, which seems to have little to do with the order of disk writes. Think about the block storage of HDFS, each time you apply for a fixed size block and the Segment here? Isn't it similar?

In addition, because the name of the index text itself is Offset as the file name, when searching, you can quickly locate the corresponding file according to the Offset you need to find, and then retrieve the content according to the file.

Therefore, the search process of Kafka is to dichotomize the file name according to the Offset to find, find the corresponding file, and then combine the contents of the sequential read area to the corresponding Offset location according to the physical address of the index metadata and the offset location of the log file.

Segment index file adopts sparse index storage mode, which reduces the size of the index file and can be operated directly in memory through mmap. Sparse index sets a metadata pointer for each corresponding Message of the data file.

It saves more storage space than dense indexes, but it takes more time to find, especially in random read scenarios, where Kafka is very inappropriate. So because of Kafka's special storage design, it also makes Kafka feel faster.

Why is Kafka stable?

I mentioned earlier why Kafka is fast. In addition to the fast feature, Kafka has other features, that is, stability.

The stability of Kafka is reflected in several dimensions:

The data is secure and can hardly be lost.

The cluster is secure and can be switched almost without awareness by Consumer in the event of failure.

High availability, even if part of the Partition is not available, the remaining Partition data still does not affect the read.

Traffic control restrictions to avoid a large number of Consumer dragging down the server bandwidth.

Current limiting mechanism

The stability of Kafka is usually determined by its overall architectural design, and many excellent features are combined to make it more excellent, such as Kafka's Qutota.

Since it is a current limit, it means that it is necessary to control the traffic bandwidth of Consumer or Producer, which usually needs to be handled on the network card, such as the common N-way switch or high-end router.

So for Kafka, it is obviously very difficult for Kafka to manipulate the network card of OS to control traffic, so Kafka adopts another special idea.

That is, if there is no way to control the amount of traffic passed by the network card, the time to return the data is controlled. For JVM programs, it's a Wait or Seelp thing.

So for Kafka, there is a set of special delay calculation rules. Kafka counts the traffic per unit time according to a window.

When the traffic exceeds the set threshold, the flow control is triggered, the current request is thrown into the Qutota Manager of Kafka, and the data is returned again when the delay time is reached.

Let's look at the methods in the ClientQutotaManager class of Kafka:

These lines of code represent the current limit calculation logic of Kafka. The general idea is as follows: suppose we set the current traffic limit not to exceed T, and calculate the current rate as O according to the window.

If O exceeds T, the speed limit will be imposed, and the speed limit will be announced as follows:

X = (O-T) / T * W

X is the time to delay, let me give you a visual example. Suppose we limit the traffic to no more than 10MB/s, and the traffic passing through within the past 5 seconds (W in the announcement, window interval) is 100MB, then the delay time is: (100-5: 10) / 10: 5 seconds.

This ensures that the size of the entire traffic will not exceed the limit after the next window is run.

You can see the delayed return of current limit through the call back code for Producer and Consumer in KafkaApis:

For the current limit of Kafka, the default is to limit the current according to client id or user, which is of little significance from the point of view of practical use. The current limit based on Topic or Partition partition level is larger than the use scenario.

Election mechanism

The meta-information behind Kafka relies heavily on Zookeeper. Again, instead of explaining Zookeeper itself, we focus on how Kafka uses ZK.

First, a picture explains Kafka's heavy dependence on ZK:

The picture comes from the Internet.

In addition to the storage of its own information using ZK, the most important thing is that Kafka uses ZK to achieve the election mechanism, in which Controller is the main introduction.

First of all, as the heart of Kafka, Controller is mainly responsible for including not limited to the following important matters:

In other words, Controller is the core role of Kafka, for Controller, using fair competition, any Broker may become Controller, ensuring the robustness of the cluster.

For Controller, the election process is as follows:

① first obtains the information of the / Cotroller node of ZK and the broker id of Controller. If the node does not exist (for example, when the cluster was first created), * the controller id obtained is-1.

② if the controller id is not-1, that is, the Controller already exists and ends the process directly.

③ if controller id is-1, it proves that Controller does not exist, and the current Broker starts to register Controller with ZK.

If the ④ registers successfully, the current Broker becomes Controller, and the onBecomingLeader () method is called to formally initialize the Controller.

Note: the Controller node is a temporary node. If the current Controller is disconnected from the Session of the ZK, the temporary node of the Controller will disappear, triggering the re-election of the Controller.

⑤ returns directly if the registration fails (Controller is created by another Broker, throws an exception, and so on).

The code can be seen directly through KafkaController:

Once the Controller is elected, other Broker will listen for changes in the ZK in response to Controller hanging in the cluster:

Thus triggering a new Controller election action. For Kafka, the whole design is very compact, the code quality is very high, and many designs can be used for reference. Similar functions are reflected in many features in Kafka, which combine to form the whole stable situation of Kafka.

How to use Kafka

Although Kafka looks very good as a whole, Kafka is not an all-powerful silver bullet, so it must have its corresponding shortcomings, so how to use Kafka, or how to use it better, needs to be realized through practical practice.

After induction and summary, the following different usage scenarios and characteristics can be found:

① Kafka is not suitable for high frequency trading system

Although Kafka has very high throughput and performance, it is undeniable that Kafka is still not as good as traditional MQ in terms of low latency of a single message. After all, MQ based on push model can gain innate advantages in the scenario of real-time message transmission.

② Kafka does not have a perfect transaction mechanism.

After 0.11, Kafka added a transaction mechanism to ensure the batch submission of Producer. In order to ensure that dirty data will not be read, Consumer can filter out inappropriate data by filtering the status of the message, but still retain the operation of reading all data.

Even so, the transaction mechanism of Kafka is still incomplete, and the main reason is that Kafka is not interested in Client, so it will not unify all common protocols, so in similar scenarios such as only being consumed once, the effect is very dependent on the implementation of the client.

The remote disaster recovery scheme of ③ Kafka is very complex.

For Kafka, if you want to achieve unaware handover across computer rooms, you need to support agents across clusters.

Because of Kafka's special append log design mechanism, the same Offset can not be reused on different Broker and different content.

That is, once the file is copied to another server, it will not be readable, compared with similar database-based MQ, it is difficult to achieve cross-cluster synchronization of data.

At the same time, it is also very difficult to reproduce Offset. It has helped customers to achieve a set of Kafka cluster Proxy across computer rooms, which has invested a lot of cost.

④ Kafka Controller architecture fails to make full use of cluster resources

Kafka Controller is similar to the decentralized idea of ES, selecting a server from the cluster as the Controller according to the election rules.

It means that changing the server not only undertakes the responsibility of Controller, but also undertakes the responsibility of Broker, resulting in the oppression of massive messages, the resources of the server can easily become the bottleneck of the cluster, resulting in the cluster resources can not be maximized.

Although Controller supports HA, it does not support distribution, which means that if you want the best performance of Kafka, each server needs to be at least the highest configuration.

⑤ Kafka does not have very intelligent partition balancing capabilities.

Usually when designing landing storage, it will be the combination of SSD and HD for hot spots or scenarios that require high performance.

At the same time, if the disk capacity of the cluster is unequal, there will be a very serious problem for Kafka. The partition generation of Kafka is calculated according to the number of Paratition, and new partitions are created on the disk with the fewest number, as shown in the following figure:

Once I helped an enterprise modify the partition creation rules, taking into account the capacity, that is, the selection of partitions according to disk capacity.

This is followed by a second problem: if a disk with large capacity has more partitions, it will cause a large number of IO to be pressed on the disk, and finally the problem will fall back to IO, which will affect the performance of other Topic on the disk.

Therefore, when considering the MQ system, we need to set the partition rules of Kafka manually.

At this point, the study of "Why Kafka is so fast" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.