Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of acker mechanism in storm, which is very detailed and has certain reference value. Friends who are interested must finish it!

First of all, let's take a look at what is meant by record-level fault tolerance. Storm allows users to assign a message id to a new source tuple when it is emitted in spout, which can be any object object. Multiple source tuple can share a message id, indicating that the multiple source tuple are the same message unit for the user. Record-level fault tolerance in storm means that storm tells the user whether each message unit has been fully processed within a specified period of time. So what is full processing, that is, the source tuple bound by the message id and the subsequent tuple generated by the source tuple are processed by every bolt that should arrive in the topology. For instance. In figure 4-1, tuple1 and tuple2 bound by message 1 in spout are processed by bolt1 and bolt2 to generate two new tuple, both of which eventually flow to bolt3. When this process is finished, message 1 is said to have been fully processed.

There is a system-level component in storm's topology called acker. The task of this acker is to track the processing paths of several tuple bound to each message id that flows out of the spout. If these tuple are not fully processed within the maximum timeout set by the user, then the acker will tell the spout that the message processing failed, and instead, it will tell the spout that the message processing was successful. In the description just now, we mentioned "recording the processing path of tuple". If you have tried to do so, you can think carefully about the complexity of the matter. But storm uses a very ingenious way to do it. Before explaining this method, let's review a mathematical theorem.

A xor A = 0.

A xor B... Xor B xor A = 0, where each Operand appears only twice.

The ingenious method used in storm is based on this theorem. The process goes like this: in spout, the system generates a corresponding 64-bit integer for the user-specified message id as a root id. The root id is passed to the acker and subsequent bolt as the unique identity of the message unit. At the same time, every time spout or bolt generates a new tuple, it will give the tuple a 64-bit integer id. After Spout launches the source tuple corresponding to a message id, it informs acker of the root id it launches and the id of those source tuple generated. Bolt, on the other hand, after receiving an input tuple for processing, will also inform acker of the id of the input tuple and the id of the newly generated tuple. Acker only needs to do a simple XOR operation on these id to determine whether the message unit corresponding to the root id has been processed or not. This process is illustrated by an illustration below.

Binding message 1 in spout generates two source tuple,id of 0010 and 1011.

Bolt1 processes tuple 0010 to generate a new tuple,id of 0110.

A new tuple,id of 0111 is generated when bolt2 processes tuple 1011.

Tuple 0110 and tuple 0111 were received in bolt3, but no new tuple was generated.

Some careful students may find that there is a possible error in the fault tolerance process, that is, if the generated tuple id is not completely different, the acker may miscalculate to 0 before the message unit is fully processed. This error does exist in theory, but in practice its probability is very low and can be ignored.

The above is all the content of the article "sample Analysis of acker Mechanism in storm". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.