Shulou(Shulou.com)06/01 Report--

This article is about how to achieve Kafka and Twitter new open source DistributedLog technology comparison, the editor feels very practical, so share with you to learn, I hope you can get something after reading this article, say no more, follow the editor to have a look.

Let's give a brief overview of Kafka and DistributedLog.

What is Kafka?

Kafka is a distributed messaging system originally opened up by Linkedin and is now managed by the Apache Software Foundation. This is a partition-based publish / subscribe system. The key concept in Kafka is Topic. There will be multiple partitions under a Topic, each with backups, distributed on different proxy servers. The producer publishes the data record to a partition under the Topic by polling or partitioning based on the primary key, while the consumer processes the data record published in the Topic.

All data is published to the master agent process of the corresponding partition, and then copied to the slave agent process, and all read data requests are processed by the master agent in turn. The slave agent is only used for redundant backups of data and is topped when the master agent can no longer provide services. The left part of figure 1 shows the data flow in Kafka.

What is DistributedLog?

Unlike Kafka, DistributedLog is not a partition-based publish / subscribe system, it is a repository of replication log streams. The key concept in DistributedLog is the continuous replication log flow. A log flow is segmented into multiple log fragments. Each log fragment is stored in Apache BookKeeper as an account in Apache BooKeeper, where the data is replicated and evenly distributed across multiple Bookie (Bookie is the storage node of Apache BookKeeper). All the data records of a log stream are sorted by the owners of the log stream, and many writing agents manage the ownership of the log stream.

Applications can also use core libraries to directly append logging. This is useful for scenarios such as replication state machines that have very high requirements for sequential and exclusive writing. Each log record appended to the end of the log stream is assigned a serial number. The reader can read the data of the log stream from any specified serial number. Read requests are also load balanced on all stored copies of that stream. The right half of figure 1 shows the data flow in DistributedLog.

What is the difference between Kafka and DistributedLog?

Because similar things have a comparable basis, we only compare Kafka partitions to DistributedLog streams in this article. The following table lists the most significant differences between the two systems.

Data model

A Kafka partition is a log stored on a proxy server disk in the form of several files. Each record is a key-value pair, but for polling data publishing, the primary key of the data can be omitted. The primary key is used to determine which partition the record will be stored on and for the log compression function. All data for a partition is stored only on several proxy servers and is replicated from the master proxy server to the slave proxy server.

A DistributedLog flow is a virtual flow that exists in the form of a series of log fragments. Each log fragment exists in the form of a BookKeeper account and is copied to multiple Bookie. Only one active log shard accepts write requests at any one time. After a specific period of time, or after the old log shard reaches the configured size (determined by the configured log sharding policy), or after the log owner fails, the old log shard will be sealed and a new log shard will be opened.

The differences in data fragmentation and distribution between Kafka partitions and DistributedLog streams determine their differences in data persistence strategies and cluster operations such as cluster expansion.

Figure 2 shows the differences between the DistributedLog and Kafka data models

Data persistence

All data in an Kafka partition is stored on a proxy server (and replicated to another proxy server). After the validity of the configuration, the data is invalidated and deleted. Alternatively, you can configure a policy to have the partition of Kafka retain the * * value of each primary key.

Similar to Kafka, DistributedLog can also configure the validity period for each flow and invalidate or delete the corresponding log fragments after the timeout. In addition, DistributedLog also provides a truncation mechanism for display. An application can explicitly truncate a log stream to a specified location of the stream. This is useful for building replicable state machines because replicable state machines need to persist state before deleting log records. Manhattan is a typical system that uses this function.

Operation

Differences in data fragmentation and distribution mechanisms also lead to differences in maintenance cluster operations, with extended cluster operations as an example.

When expanding a Kafka cluster, existing partitions are usually redistributed. The redistribution operation moves the Kafka partition to different replicas to achieve a balanced distribution. This involves copying the data of the entire stream from one copy to another. We have also said many times that you must be very careful when performing redistribution operations to avoid running out of disk and network resources.

Extending the DistributedLog cluster works in a very different way. DistributedLog consists of two layers: the storage layer (Apache BooKeeper) and the service layer (write and read agents). When extending the storage tier, we just need to add more Bookie. The new Bookie is immediately discovered by the write agent and immediately used to write new log shards. There is no redistribution when you extend the data storage tier. There is a redistribution operation only when the service layer is added, but this redistribution is only the sovereignty of the mobile log flow so that the network generation width can be evenly distributed among the agents. This redistribution process is only related to sovereignty, and there is no data migration operation. This isolation of the storage layer and the service layer not only makes the system have the mechanism of automatic expansion, but also allows different types of resources to expand independently.

Writing and producer

As shown in figure 1, the Kafka producer writes the data in batches to the master proxy server in the Kafka partition. The slave proxy server in the ISR (synchronous replication) collection copies the records away from the master agent. A record is considered to have been written successfully only after the master agent has received a successful response from all copies of the ISR collection. You can configure producers to wait only for responses from the master agent or for responses from all agents in the ISR collection.

In DistributedLog, there are two ways to write data to the DistributedLog stream, either by using a Thrift thin client through a write agent (known as multiple writes), or by directly interacting with the storage node through DistributedLog's core library (known as individual writes). * is suitable for building a message system, and the second is suitable for building a replication state machine. You can check the relevant sections of the DistributedLog documentation for more information and references to find out the way you need it.

The owner of the log flow concurrently writes a batch of records to the Bookie in the form of BookKeeper entries and waits for the Quorum results of multiple Bookie. The size of the Quorum depends on the ack_quorum_size parameter of the BookKeeper account and can be configured to the level of the DistributedLog stream. It provides persistence flexibility similar to that of Kafka producers. We will compare more differences in replication algorithms between the two in the next "replication" section.

Both Kafka and DistributedLog support end-to-end batch operations and compression mechanisms. But one subtle difference between the two is that writes to DistributedLog are flushed to the hard disk via fsync before receiving a response, and we don't find that Kafka provides a similar reliability guarantee.

Reading with consumers

The Kafka consumer reads the data record from the master proxy server. The premise of this design is that the data on the master agent is still in the file system page cache in most cases. This is a good idea in terms of making full use of the file system page cache and achieving high performance.

DistributedLog takes a completely different approach. Because there is no clear master-slave relationship between each storage node, DistributedLog can read data from any storage node that stores the relevant data. In order to achieve a predictable low latency, DistributedLog introduces an inference read mechanism, that is, after the configured read operation time limit is exceeded, it will try to get data again on different replicas.

This may cause higher read pressure on the storage node than Kafka. However, if the read timeout is configured so that 99% of the storage nodes do not time out, the latency problem can be greatly solved, resulting in only 1% additional read pressure.

The difference in the consideration and mechanism of reading is mainly due to the difference between the replication mechanism and the Istroke O system of the storage node, which will be discussed later.

Copy

Kafka uses the ISR replication algorithm: choose a proxy server as the primary. All writes are sent to the master agent, and all slave agents in the ISR collection read and copy data from the master agent. The main agent maintains a high water level line (HW,High Watermark), that is, the offset of the data records submitted by each partition. The high water mark is constantly synchronized to the slave agent, and checkpoints are periodically recorded on all agents for recovery. The high water mark is not updated by the master agent until the replicas in all ISR collections write data to the file system (not necessarily on disk) and send a response back to the master agent.

The ISR mechanism allows us to increase or decrease the number of replicas, making a tradeoff between availability and performance. However, the side effect of expanding or shrinking the collection of replicas is to increase the likelihood of data loss.

DistributedLog uses the Quorum voting replication algorithm, which is common in consistency algorithms such as Zab, Raft, and Viewstamped Replication. The owner of the log flow writes the data records to all storage nodes concurrently, and the data is considered to have been submitted successfully only after being confirmed by the vote of more than the configured number of storage nodes. The storage node also responds to write requests only after the data is explicitly brushed to disk by the flush operation.

The owner of the log stream will also maintain the offset of the submitted data records of the log stream, which is known as LAC (LastAddConfirmed) in Apache BookKeeper. The LAC is also saved in the data record (to save additional RPC call overhead) and is constantly copied to other storage nodes. The size of the replica collection in DistributedLog is configurable at each log shard level of each flow. Changing the replication parameters will only affect the new log fragments, not the existing ones.

Storage

Each Kafka partition is saved on the agent's disk in the form of several files. It uses the page cache of the file system and the Imax O scheduling mechanism to achieve high performance. Kafka also makes use of Java's sendfile API to write and read data efficiently from the agent. However, in some cases (such as untimely processing by consumers, random reads and writes, etc.), the data in the page cache is eliminated frequently, and its performance is highly uncertain.

DistributedLog uses a different Icano model. Figure 3 shows the Bookie (the storage node of BookKeeper)'s Imax O mechanism. Write (blue line), end read (red line), and intermediate read (purple line) are three common Imax O operations that are isolated into three physically different Imax O subsystems. All writes are sequentially appended to the log files on disk and then submitted in bulk to the hard disk. After the writes are persisted to disk, they are put into a Memtable and the response is sent back to the client.

The data in the Memtable is asynchronously refreshed to the cross-access index data structure: the record is appended to the log file, and the offset is indexed according to the record ID in the index file of the ledger. The data of * must be in Memtable and can be used for end read operations. Intermediate reads get data from the log file. Due to the existence of physical isolation, Bookie nodes can make full use of the network inflow bandwidth and the sequential writing characteristics of the disk to meet the write request, as well as the network outflow generation width and the IOPS processing capacity provided by multiple disks to meet the read request without interfering with each other.

The above is how to achieve the new open source DistributedLog technology comparison between Kafka and Twitter. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.