Shulou(Shulou.com)05/31 Report--

This article shows you how to understand Storm's performance test report. It is concise and easy to understand. It will definitely make your eyes shine. I hope you can learn something from the detailed introduction of this article.

During an interview, a CTO asked in detail about testing machine performance, as follows:

The configuration is as follows:

1 CPU 8 cores ×2 machines

2 Main frequency 2.26

3 Operating system redhat

4 network kilobytes Ethernet

5 Storm version number: 0.6.1

test method

Storm is a stream processing system. It takes tuple as the basic unit, and each tuple can contain multiple fields. We define two fields for tuple:

Data: Store the original data, here is 1000 bytes of data, in this test we are only directly forwarding data, so the only processing overhead is 1000 bytes of memory copy

ltsInfo: timestamp information. After each processing module, we add the timestamp at that time to this field. Finally, the statistical module can calculate the total delay according to this time information. Since the timestamps of different machines are not synchronized, this introduces inherent errors in computational delay, and the solution is to put the data transmission module and the final statistics module on a physical machine.

A note on testing storm on a distributed cluster: on storm, it is difficult to specify which physical machine a component runs on. Storm automatically distributes tasks evenly among the machines in the cluster, so in testing we will scale storm the way it works, rather than designing atypical scenarios (assigning a specific machine to a component to run on, thus breaking the principle of equal distribution).

That is to say, the mechanism of scouring test is adopted to deal with it

memory usage

Taobao's test result is about 20,000 pieces/second.

In my local machine test is more than 25,000, the difference may be Taobao in the process of testing some time TimeStamp.

No new machines have been added so far to allow for some lateral expansion. There is no comparison of how much computing power the newly added machines have added to the system cluster.

Please refer to the specific test method

Amoy test connection:

http://blog.linezing.com/? p=1048&replytocom=828

In our real-world tests, GC will cause a relatively large change in the speed of data processing. Tuple backlogs are a particularly common phenomenon.

test conclusion

After the above tests we can conclude the following:

stormThe processing capacity of a single pipeline is approximately 20000 tupe/s, (each tuple size is 1000 bytes)

Storm system has a millisecond processing delay

Horizontal scaling in clusters can increase the processing capacity of the system, and the measured result is 1.6 times.

Storm uses a large number of threads, even if a single processing pipeline system, there are more than a dozen threads running at the same time, so almost all 16 CPUs are running, load average is about 3.5

Jvm GC generally has a limited impact on system performance, but when memory is tight, GC will become a bottleneck in system performance.

The performance of using external processors is obviously degraded, so under high performance requirements, try to use storm built-in processing mode.

The test tool is used: nmon

That's how to understand Storm's performance test report. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserves, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.