Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what is the ack mechanism of Storm". In the daily operation, I believe that many people have doubts about what the ack mechanism of Storm is. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "what is the ack mechanism of Storm?" Next, please follow the editor to study!

Ack mechanism

Storm ensures that every data sent from Spout is fully processed, and data sent from Spout may produce thousands of data. A Tuple is fully processed means that the Tuple and all Tuple generated by the Tuple are processed successfully. A Tuple is considered to be a failure to process within the timeout time (including the displayed fail and the failure caused by the timeout). This timeout time can be set through Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS. The default duration of Timeout is 30 seconds.

Storm's Bolt has BsicBolt and RichBolt. In BasicBolt, BasicOutputCollector is automatically associated with the input tuple when emit data, and the input tuple is automatically ack at the end of the execute method.

To implement ack when using RichBolt, you need to display the source tuple of the specified emit data, that is, collector.emit (oldTuple, newTuple); and you need to call the ack of the source tuple to ack after the execute execution is successful.

It is important to note that to implement the ack mechanism, the messageId must be specified when the spout emits the tuple. And you need to cache the tuple in spout, delete the tuple of ack from the cache queue, and choose to resend the tuple of fail. Different Tuple can be bound to the same messageId, indicating that the multiple Tuple are the same message unit for the user.

This messageId is just for the convenience of our business to distinguish which Tuple is returned, and it is not handled internally by Storm. Therefore, when different Tuple binds the same messageId, you can't tell which Tuple succeeded or failed in ack and fail, only the messageId bound to it.

Adjust Reliability (Tuning Reliability)

Acker task is very lightweight, so you don't need much acker in a topology. You can track its performance through Strom UI (id:-1). If its throughput doesn't look normal, then you need to add more acker.

If reliability is not that important to you-you don't care about losing some data in some cases of failure, then you can get better performance by not tracking these tuple trees. Not tracking messages will halve the number of messages in the system because an ack message is sent for each tuple. And it needs less id to save the downstream tuple and reduce the bandwidth consumption.

There are three ways to remove reliability. The first is to set Config.TOPOLOGY_ACKERS to 0. 0. In this case, storm calls the ack method of spout immediately after spout emits a tuple. This means that the tuple tree will not be tracked.

The second approach is to remove reliability at the tuple level. You can not specify messageid when launching tuple to achieve the goal of not following a specific spout tuple of rice dumplings.

Finally, if you are not very concerned about the success of some part of a tuple tree, you can unanchor these tuple when you launch them. In this way, the tuple is not in the tuple tree and will not be tracked.

Ack principle

There is a special task in Storm, and they are responsible for tracking the Tuple tree of every Tuple sent out by spout. When acker finds that a Tuple tree has been processed, it sends a message to the task that generated the Tuple. Acker's tracking algorithm is one of the main breakthroughs of Storm. For any large Tuple tree, it only needs a constant 20 bytes to track.

The principle of Acker tracking algorithm: acker saves a check value of ack-val for each spout-tuple, its initial value is 0, and then every time a Tuple or Ack is emitted a Tuple, the id of this Tuple will be XOR with this check value, and the resulting value will be updated to the new value of ack-val. So if every Tuple emitted is ack, then the final value of ack-val must be 0. Acker determines whether it is fully processed based on whether the ack-val is 0, and if it is 0, it is considered to have been fully processed.

At this point, the study on "what is the ack mechanism of Storm" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.