Shulou(Shulou.com)05/31 Report--

The main content of this article is to explain "what is the optimization idea of storm topology". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the idea of storm topology optimization?"

Suppose an environment, three machines, cpu core 4jinnet KM net

Topology

Componentnameexecutor latency (ms) spouts-boltA0.5boltB0.1boltC0.5boltD1.0boltE0.2

Executor represents the time when the data enters the execution of the bolt execute method, and storm is obtained by sampling. There will certainly be an error in the total number of samples, but for the time being, we think that it is certain, which is equivalent to randomly selecting one out of 1W 1s, so we assume that the executor latency is credible for the time being. Finally, we verify our hypothesis through the actual efficiency.

Suppose that we submit the topology with one parameter, and the ui looks at the parameters such as the table after running. We observe that two bolt delays at 0.5ms, B bolt delay at 0.1ms, D bolt delay at 1.0ms, E bolt delay at 0.2ms, so we can think of performance: B=2E=5A=5C=10D, so we can improve the efficiency of the whole topology and adjust the parallelism of each bolt. If we set the parallelism of B to X, we can get X+2X+5X*2+10X=All executor. If there is a blocking condition in a bolt, we can increase the parallelism a little bit more. For the three machines in this environment, the maximum parallelism is 3, 4, 12, X "= 1, so it can be concluded that the parallelism of B bolt is 1, so the processing efficiency per second is about 1, 000, 0.1, 1, 1W per second.

In addition, each worker in storm has a thread pool, and the pool size defaults to the number of core. We can increase the pool size and cpu utilization according to the situation.

Other parameters will not be mentioned here, and can be adjusted according to the actual situation.

At this point, I believe you have a deeper understanding of "what is the idea of storm topology optimization?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.