Shulou(Shulou.com)06/03 Report--

Hyperledger Fabric recommends Kafa for production environments. Kafa is a distributed log system with horizontal scalability and crash tolerance. There can be multiple Kafka nodes in the Hyperledger Fabric blockchain, which can be managed synchronously using zookeeper. This paper will introduce the basic working principle of Kfaka and the principle of realizing consensus by using Kafka and zookeeper in HyperledgerFabric, and analyze the process of reaching Kafka consensus in Hyperledger Farbic through an example.

If you want to quickly grasp the chain code and application development of Fabric block chain, it is recommended to visit Huizhi.com 's online interactive course:

Fabric Block chain Java Development detailed explanation Fabric Block chain NodeJs Development detailed explanation 1. Kafka working principle

Kafka is essentially a message processing system that uses the classic publish-subscribe model. The consumer of the message subscribes to a specific topic to receive notification of the new message, and the producer is responsible for publishing the message.

As the data size of a topic becomes larger and larger, it can be split into multiple partitions, and Kafka ensures that messages within a partition are arranged in order.

Kafka does not track which messages are read by consumers, nor does it automatically delete messages that have been read. Kafka saves messages for a period of time, such as a day, or until the data size exceeds a certain threshold. Consumers need to poll for new messages, so they can locate messages according to their needs, so they can replay or reprocess events. Consumers are in different consumer groups, corresponding to one or more consumer processes. Each partition is decibel to a single consumer process, so the same message is not read multiple times.

The crash fault tolerance mechanism is achieved by copying partitions between multiple Kafka agents. So if an agent dies due to a software or hardware failure, the data will not be lost. Of course, a leader-follow mechanism is also needed, in which the leader holds the partition and the follower replicates the partition. When a leader dies, a follower becomes a new leader.

If a consumer subscribes to a principal, how does it know from which partition leader to read the subscription message?

The answer lies in zookeeper services.

Zookeeper is a distributed key-value repository that is commonly used to store metadata and implement clustering mechanisms. Zookeeper allows clients of the service (Kafka proxy) to subscribe to changes and be notified in real time. This is why the agent determines which partition leader should be used. Zookeeper has super fault tolerance, so the operation of Kafka depends heavily on it.

The metadata stored in zookeeper includes:

Consumer group read offset access control list for each partition, used to access authorization and limit producer and consumer quotas, maximum number of messages per second, partition leader and health information 2. Kafka in Hyperledger Fabric

To understand how Kafka works in superbook Hyperledger Fabric, you first need to understand several important terms:

Chain-refers to a set of log Channel that clients (channel / channel) can access-a channel is similar to a topic, where authorized peer nodes (peer) can subscribe and become members of the channel. Only members of the channel can trade on the channel, and transactions in one channel cannot be seen in other channels. OSN-the sort service node (Ordering Service Node), which is called the sort node in Fabric. The sorting node is responsible for: customer authentication allows clients to filter and verify configuration transactions through a simple interface to write or read channels, reconfigure channels or create new channels RPC-remote procedure call (Remote Procedure Call), which is a communication protocol for calling services on other machines without knowing the communication and implementation details The purpose is to call the function broadcast PRC-transaction submission call on other machines in the network just like calling the local function. The sorting node executes the distribution RPC-transaction distribution request, and when the transaction is processed by the kafka agent, it is distributed to the request node.

Note that although Kafka is called Consensus in Hyperledger Fabric, its core is transaction sorting services and additional crash tolerance.

The actual running logic of Kafka in Hyperledger Fabric is as follows:

For each chain, there is a corresponding partition and each chain corresponds to a single partition topic sorting node responsible for relaying transactions from a particular chain (received via broadcast RPC) to the corresponding partition sorting node to read the partition and obtain a list of sorted transactions agreed among all sorting nodes. transactions in a chain are processed in batches at a fixed time, that is, when the first transaction of a new batch comes in. Start timing: when the transaction reaches the maximum number or times out, the batch is split to generate a new block scheduled transaction is another transaction, each sorting node is generated by the timer above to maintain a local log for each chain, and the generated block is saved in the local ledger. The transaction block is returned to the client by distributing the RPC. When a crash occurs, you can use different sorting nodes to distribute the block. Because all sorting nodes maintain local logs

III. Analysis of Hyperledger Fabric Kafka examples

Consider the following figure, assuming that the sorting nodes OSN0 and OSN2 are connected to the broadcast client and OSN1 is connected to the distribution client.

OSN0 already has a transaction foo, which is relayed to the kafka cluster. At this time, OSN2 broadcasts the transaction baz to the cluster. Finally, the transaction bar is sent to the cluster by OSN0. Now there are three transactions. You can see in the figure that the location offset client of the three transactions sends a distribution request. In the local log of OSN1, the above three transactions are in the transaction block. So OSN1 returns the blocks to the client, and the processing ends.

The high performance of Kakfa is of great help to Hyperledger Fabric. Multiple sorting nodes are synchronized through Kafka, while Kafka itself is not a sorting node, it just connects the sorting nodes through streams. Although Kafka supports crash tolerance, it does not provide protection against malicious attacks on the network. A Byzantine fault tolerance scheme (BFT) is needed to combat malicious attacks, but this mechanism has yet to be implemented in the Farbic framework.

All in all, in Hyperledger Farbic, the Kafka consensus module can be used in a production environment, and it can support crash tolerance, but not against malice.

Original: The ABCs of Kafka in Hyperledger Fabric

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.