Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you how to understand the kafka consensus of Fabric blockchain. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Hyperledger Fabric recommends Kafa for production environments. Kafa is a distributed log system with horizontal scalability and crash tolerance. There can be multiple Kafka nodes in the Hyperledger Fabric blockchain, which can be managed synchronously using zookeeper. The following will introduce the basic working principle of Kfaka, and the principle of using Kafka and zookeeper to achieve consensus in Hyperledger Fabric, and analyze the process of reaching Kafka consensus in Hyperledger Farbic through an example.

I. the working principle of Kafka

Kafka is essentially a message processing system that uses the classic publish-subscribe model. The consumer of the message subscribes to a specific topic to receive notification of the new message, and the producer is responsible for publishing the message.

As the data size of a topic becomes larger and larger, it can be split into multiple partitions, and Kafka ensures that messages within a partition are arranged in order.

Kafka does not track which messages are read by consumers, nor does it automatically delete messages that have been read. Kafka saves messages for a period of time, such as a day, or until the data size exceeds a certain threshold. Consumers need to poll for new messages, which allows them to locate messages according to their needs, so they can replay or reprocess events. Consumers are in different consumer groups, corresponding to one or more consumer processes. Each partition is decibel to a single consumer process, so the same message is not read multiple times.

The crash fault tolerance mechanism is achieved by copying partitions between multiple Kafka agents. So if an agent dies due to a software or hardware failure, the data will not be lost. Of course, a leader-follow mechanism is also needed, in which the leader holds the partition and the follower replicates the partition. When a leader dies, a follower becomes a new leader.

If a consumer subscribes to a principal, how does it know from which partition leader to read the subscription message?

The answer lies in zookeeper services.

Zookeeper is a distributed key-value repository that is commonly used to store metadata and implement clustering mechanisms. Zookeeper allows clients of the service (Kafka proxy) to subscribe to changes and be notified in real time. This is why the agent determines which partition leader should be used. Zookeeper has super fault tolerance, so the operation of Kafka depends heavily on it.

The metadata stored in zookeeper includes:

Read offset of consumer grouping in each partition

Access control list for access authorization and restriction

Producer and consumer quota, maximum number of messages per second

Regional leaders and health information

2. Kafka in Hyperledger Fabric

To understand how Kafka works in superbook Hyperledger Fabric, you first need to understand several important terms:

Chain-refers to a set of logs that can be accessed by clients (channel / channel)

Channel-A channel is similar to a topic, where authorized peer nodes (peer) can subscribe and become members of the channel. Only members of the channel can trade on the channel, and transactions in one channel cannot be seen in other channels.

OSN-the sort service node (Ordering Service Node), which is called the sort node in Fabric. The sort node is responsible for:

Conduct customer authentication

Allows clients to write or read channels through a simple interface

Perform filtering and verification of configuration transactions to reconfigure channels or create new channels

RPC-remote procedure call (Remote Procedure Call) is a communication protocol used to call services on other machines without knowing the communication and implementation details. The purpose is to call functions on other machines in the network just like local functions.

Broadcast PRC-transaction submission call, executed by the sort node

Distribute RPC-deal distribution request, which is distributed to the requesting node when the transaction is processed by the kafka agent

Note that although Kafka is called Consensus in Hyperledger Fabric, its core is transaction sorting services and additional crash tolerance.

The actual running logic of Kafka in Hyperledger Fabric is as follows:

For each chain, there is a corresponding partition

Each chain corresponds to a single partition theme

The sorting node is responsible for relaying transactions from a particular chain (received via broadcast RPC) to the corresponding partition.

The sorting node can read the partition and get a list of sorted transactions that agree among all sorting nodes.

Transactions in a chain are processed in batches on a regular basis, that is, when the first transaction of a new batch comes in, timing begins.

Batch segmentation is performed when the transaction reaches the maximum number or after the timeout to generate a new block

A timed transaction is another transaction, generated by the timer described above

Each sorting node maintains a local log for each chain, and the resulting blocks are saved in the local ledger.

The transaction block returns to the client by distributing the RPC

When a crash occurs, you can use different sort nodes to distribute blocks, because all sort nodes maintain local logs

III. Analysis of Hyperledger Fabric Kafka examples

Consider the following figure, assuming that the sorting nodes OSN0 and OSN2 are connected to the broadcast client and OSN1 is connected to the distribution client.

OSN0 already has a transaction foo, which is relayed to the kafka cluster

At this point, OSN2 broadcasts the transaction baz to the cluster

Finally, the transaction bar is sent to the cluster by OSN0

The cluster now has three transactions, and you can see the location offset of the three transactions in the log in the figure.

The client sends a distribution request, and in the local log of OSN1, the above three transactions are in the delivery block.

So OSN1 returns the blocks to the client, and the processing ends.

The high performance of Kakfa is of great help to Hyperledger Fabric. Multiple sorting nodes are synchronized through Kafka, while Kafka itself is not a sorting node, it just connects the sorting nodes through streams. Although Kafka supports crash tolerance, it does not provide protection against malicious attacks in the network. A Byzantine fault tolerance scheme (BFT) is needed to combat malicious attacks, but this mechanism has yet to be implemented in the Hyperledger Farbic framework.

All in all, in Hyperledger Farbic, the Kafka consensus module can be used in a production environment, it can support crash fault tolerance, but can not resist malicious attacks.

This is how the Fabric blockchain kafka consensus shared by the editor is understood. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.