Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to improve the running speed of storm. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

(1) the algorithm in the system should be tuned. It is possible that an algorithm wastes a small amount of time, but because the amount of data may be so large, it may waste a lot of time in 1 second as a whole. Therefore, the design of the algorithm is still more important.

(2) secondly, it is to adjust the spout and bolt which take up more resources and the operation speed is relatively slow. When designing a topology, you need to design the parallelism of each bolt. For slow bolt, we need to increase their parallelism, so that more resources are needed for these calculations. Here, the running speed of bolt can be seen in the ui interface, as shown below:

As shown in the figure above, capacity represents a capacity, which is actually the percentage of resources occupied. For example, 0.799 means that 79.9% of the resources allocated to this bolt are occupied. The higher the number, the slower the processing speed and the more parallelism.

(3) then set the number of acker. Acker is the thread that makes the ack call after bolt has successfully processed it (or the process, I forgot). When the amount of data is large, the number of times you need to use this thread is more, so it is possible that this thread is the factor restricting the processing speed. Therefore, you can appropriately increase the number of acker to make calls to ack. In the system, if not set, the number of acker defaults to 1; you can set it in topology with the following statement:

Conf.put (Config.TOPOLOGY_ACKER_EXECUTORS, 10); / / set the number of acker

(4) when the amount of data in the cluster is relatively large, it is best to set the amount of data waiting to be processed in spout. When the amount of data waiting in the cluster is relatively large, that is, the data is sent faster, but the processing is too slow. At this time, you should prevent the sending of spout, otherwise it may cause the system queue to burst. Therefore, set the following:

Conf.put (Config.TOPOLOGY_MAX_SPOUT_PENDING, 10000); / / set the maximum number of unprocessed tuple (no ack/failed) replies on a spout task to prevent the tuple queue from exploding.

(5) when Spout calls the nextTuple method, if there is no emit tuple, you need to hibernate 1ms by default. This specific policy is configurable, so you can set it according to your specific scenario to make rational use of cpu resources.

Topology.spout.wait.strategy "backtype.storm.spout.SleepSpoutWaitStrategy" topology.sleep.spout.wait.strategy.time.ms 1 on "how to improve the running speed of storm" this article is shared here, I hope the above content can be of some help to you, so that you can learn more knowledge, if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.