Shulou(Shulou.com)06/03 Report--

Author: sun Mengyao | Meituan Dianping

The main content of this paper: by comparing the performance of distributed real-time computing framework Flink and Storm, it provides data reference for Meituan Dianping real-time computing platform and business.

one。 Background

Apache Flink and Apache Storm are two distributed real-time computing frameworks widely used in the industry. Among them, Apache Storm (hereinafter referred to as "Storm") has been widely used in Meituan Dianping's real-time computing business (refer to the reliability assurance test of Storm). There are management platforms, commonly used API and corresponding documents, and a large number of real-time jobs are built based on Storm. Apache Flink (hereinafter referred to as "Flink") has attracted much attention recently, with the characteristics of high throughput, low latency, high reliability and accurate computing, and has a good support for the event window. At present, it has been applied in Meituan Dianping's real-time computing business.

In order to familiarize ourselves with the Flink framework, verify its stability and reliability, evaluate its real-time processing performance, identify shortcomings in the system, find its performance bottleneck and optimize it, and provide users with the most suitable real-time computing engine, we have carried out a series of experiments to test the performance of the Flink framework compared with the experienced Storm framework. Computing Flink as a real-time computing framework to ensure the semantics of "at least once" and "just once" consumes resources. This paper puts forward suggestions and data support for the decision-making of real-time computing platform resource planning, framework selection, performance tuning and the construction of Flink platform, and provides some reference for subsequent SLA construction.

Comparison of Flink and Storm frameworks:

Cdn.xitu.io/2019/5/30/16b08bf39238aa20?w=640&h=485&f=png&s=190685 ">

two。 Test target

Evaluate the current performance of Flink and Storm real-time computing frameworks under different scenarios and different data pressures, obtain their detailed performance data and find the limits of processing performance; understand the impact of different configurations on Flink performance, analyze the applicable scenarios of various configurations, and get tuning recommendations.

2.1 Test scenario "input-output" simple processing scenario

Through the test of dealing with logical scenarios as simple as "input-output", the interference of other factors is reduced as much as possible, and the performance of the two frameworks themselves is reflected.

At the same time, measuring the limits of the framework's processing power, the performance of handling more complex logic is no higher than that of pure "input-output".

Scenarios where the user's job takes a long time

If the user's processing logic is complex, or if the user accesses external components such as a database, the execution time will increase, and the performance of the job will be affected. Therefore, we test the scheduling performance of the two frameworks in scenarios where user jobs take a long time.

Window statistics scene

In real-time computing, there is often a need to count the time window or count window, such as the number of visits every five minutes in a day, how many out of every 100 orders use discounts, and so on. Flink is more powerful than Storm in window support and API is more complete, but we also want to know the performance of the two frameworks in the common scenario of window statistics.

Accurate calculation of the scenario (that is, the message delivery semantics is "exactly once")

Storm can only guarantee the message delivery semantics of "at most once" (At Most Once) and "at least once" (At Least Once), that is, there may be duplicates. There are many business scenarios that require high accuracy of data. We hope that the delivery of messages will not be repeated or omitted. Flink supports the semantics of "just once" (Exactly Once), but under limited resource conditions, stricter accuracy requirements can lead to higher costs, which can affect performance. Therefore, we test the performance of the two frameworks under different message delivery semantics, hoping to provide data reference for the resource planning of accurate computing scenarios.

2.2 performance Metrics Throughput (Throughput) the number of data successfully transmitted by the computing framework per unit time. The throughput of this test is measured in bars per second. It reflects the load capacity of the system, and how much data the system can handle per unit time under the corresponding resource conditions. Throughput is often used in resource planning, and it is also used to analyze system performance bottlenecks, so as to adjust resources to ensure that the system can achieve the processing capacity required by users. Assuming that the merchant can make 20 lunches per hour (20 pieces per hour), and a delivery boy can only deliver two pieces per hour (2 pieces per hour), the bottleneck of this system lies in the delivery link of the brother. Ten takeout brothers can be arranged to deliver to the merchant. Delay (Latency) the time it takes for data to enter and flow out of the system. The delay in this test is in milliseconds. It reflects the real-time processing of the system. A large number of real-time computing services such as financial transaction analysis have higher requirements for delay, and the lower the delay is, the stronger the real-time data is. Suppose it takes 5 minutes for a merchant to make lunch and 25 minutes for delivery, and users feel a 30-minute delay in this process. If the delay becomes 60 minutes after changing the delivery plan, and the food gets cold by the time it is delivered, the new plan is unacceptable. three。 Test environment

Set up a Standalone cluster composed of 1 master node and 2 slave nodes for Storm and Flink respectively for this test. In order to observe the performance of Flink in the actual production environment, some of the test contents are also tested in on Yarn environment.

3.1 Cluster parameters

3.2 frame parameters

four。 Test method 4.1 Test flow

Data production

DataGenerator generates data at a specific rate and writes it to a Topic (TopicData) of Kafka with self-increasing id and eventTime timestamps.

Data processing.

Storm Task and Flink Task (different for each test case) start consuming from the same Offset of Kafka Topic Data, and write the result and the corresponding inTime and outTime timestamps into the two Topic (Topic Storm and Topic Flink), respectively.

Index statistics

MetricsCollector counts the test metrics from the two Topic according to the time window of outTime, and writes the corresponding metrics into the MySQL table every five minutes.

MetricsCollector takes a five-minute rolling time window according to outTime, calculates the average throughput of five minutes (the number of output data), the median delay within five minutes (outTime-eventTime or outTime-inTime) and 99 lines, and writes them into the corresponding data table of MySQL. Finally, the average value of the throughput in the MySQL table is calculated, the median delay and delay 99 lines are selected, and the image is drawn and analyzed.

The default parameters Storm and Flink are At Least Once semantics by default.

The number of ACK,ACKer enabled by Storm is 1. The Checkpoint interval for Flink is 30 seconds, and the default StateBackend is Memory. Ensure that Kafka is not a performance bottleneck and eliminate the impact of Kafka on test results as much as possible. During the test delay, the data production rate is less than the data processing capacity, assuming that the data is read immediately after it is written into the Kafka, that is, the eventTime is equal to the time the data enters the system. When testing throughput, read from the oldest of the Kafka Topic, assuming that there is sufficient test data in that Topic. The test case IdentityIdentity use case mainly simulates the input-output simple processing scenario, reflecting the performance of the two frameworks themselves. The input data is "msgId,eventTime", where eventTime is regarded as the time when the data was generated. A single input data is about 20B. Record inTime when you enter the job processing process, and record outTime when the job processing is completed (when you are ready for output). After the job reads the data from Kafka Topic Data, it appends a timestamp to the end of the string and then outputs it directly to Kafka. The output data is "msgId,eventTime, inTime, outTime", and a single output data is about 50B.

Sleep

Sleep use cases mainly simulate scenarios where user jobs take a long time, reflect the weakening of framework differences caused by complex user logic, and compare the scheduling performance of the two frameworks.

The input and output data are the same as Identity.

After reading the data, wait for a certain amount of time (1 ms), append a timestamp to the end of the string and output it.

The WindowedWord CountWindowedWord Count use case mainly simulates window statistics scenarios, reflecting the performance differences between the two frameworks in window statistics. In addition, it is also used to test the accurate calculation scenario, which reflects the performance of Flink exactly once delivery. The input is in JSON format, including msgId, eventTime, and a sentence consisting of several words separated by spaces. A single input data is about 150 B. After reading the data, parse the JSON, then divide the sentence into corresponding words, send it to CountWindow with eventTime and inTime timestamps for word counting, record the largest and smallest eventTime and inTime in a window, and finally output to the corresponding Topic of Kafka with outTime timestamps. The degree of concurrency of Spout/Source and OutputBolt/Output/Sink is always 1. When the degree of concurrency increases, only the concurrency of JSONParser and CountWindow increases. Because of Storm's weak support for window, CountWindow is implemented manually using a HashMap, and Flink uses native CountWindow and corresponding Reduce functions.

five。 Test results 5.1 Identity single Thread Throughput

In the figure above, the blue column is the throughput of single-thread Storm jobs, and the orange column is the throughput of single-thread Flink jobs. Under Identity logic, Storm single-thread throughput is 87,000 entries per second, while Flink single-thread throughput can reach 350,000 entries per second. When the Partition number of Kafka Data is 1, the throughput of Flink is about 3.2 times that of Storm, and when its Partition number is 8, the throughput of Flink is about 4.6 times that of Storm.

It can be seen that the throughput of Flink is about 3-5 times that of Storm.

5.2 Identity single-thread job delay

Using outTime- eventTime as the delay, the blue broken line is Storm and the orange broken line is Flink. The dashed line is 99 lines and the solid line is the median. It can be seen from the figure that as the amount of data increases, the delay of Identity increases gradually. The increasing speed of line 99 is faster than that of median, and that of Storm is faster than that of Flink. Among them, the test data of QPS above 80000 exceeds the throughput capacity of Storm single thread, so it is impossible to test Storm, only the curve of Flink. Comparing the data at the rightmost end of the broken line, we can see that when the Storm QPS approaches throughput, the median delay is about 100ms, the 99 line is about 700ms, the Flink median is about 50ms, the 99 line is about 300ms. The delay of Flink in full throughput is about half of that of Storm. 5.3 Sleep Throughput

As can be seen from the figure, the throughput of Storm and Flink single thread is about 900s / s at Sleep1 milliseconds, and increases linearly as concurrency increases. Comparing the blue and orange columns, it can be found that the throughput of the two frames is basically the same. 5.4 Sleep single-threaded job latency (median)

OutTime-eventTime is still used as the delay, and you can see from the figure that the latency of Flink is still lower than that of Storm when Sleep is 1 millisecond. 5.5 Windowed Word Count single-threaded throughput

A single thread executes a count window with a size of 10, and the throughput statistics are shown in the figure. As can be seen from the figure, the throughput of Storm is about 12,000 / s, and that of Flink Standalone is about 43,000 / s. The throughput of Flink was still more than 3 times that of Storm. 5.6 Windowed Word Count Flink At Least Once versus Exactly Once Throughput

Because the processing speed of multiple parallel tasks of the same operator may be different, the contents of different snapshots in the upstream operator may be included in the same snapshot when they reach the downstream operator after the processing of the intermediate parallel operator. In this way, this part of the data will be processed repeatedly. Therefore, Flink needs to be aligned under Exactly Once semantics, that is, before all the data in the current earliest snapshot is processed, the data belonging to the next snapshot is not processed, but waits in the cache. In the current test cases, alignment is required between JSON Parser and CountWindow, CountWindow and Output, and there is some consumption. In order to reflect the alignment scene, the concurrency degree of Source/Output/Sink is still 1, which improves the concurrency of JSONParser/CountWindow. For details of the process, see the Windowed Word Count flow chart above. In the figure above, the orange column is the throughput of AtLeast Once, and the × × column is the throughput of Exactly Once. Comparing the two, we can see that under the current concurrency conditions, the throughput of Exactly Once is 6.3% lower than that of At Least Once. 5.7 Windowed Word Count Storm At Least Once and At Most Once Throughput comparison

After Storm sets the number of ACKer to zero, each message automatically ACK when it is sent, no longer waiting for the ACK of Bolt, nor resending the message, which is At Most Once semantics. In the figure above, the blue column is the throughput of At Least Once, and the light blue column is the throughput of At Most Once. Comparing the two, we can see that under the current concurrency condition, the throughput under At Most Once semantics is 16.8% higher than that of At Least Once. 5.8 Windowed Word Count single-threaded job latency

Both entity and Sleep observe outTime-eventTime, because the processing time of the job is short or the accuracy of Thread.sleep () is low, outTime-inTime is zero or has no comparative meaning; the value of outTime-inTime can be effectively measured in Windowed Word Count and drawn on the same picture as outTime-eventTime, where outTime-eventTime is dashed line and outTime-InTime is solid line. By observing the two orange broken lines, we can find that the delay calculated by Flink in both ways remains at a low level; by observing the two blue curves, we can find that Storm has a lower outTime-inTime and a higher outTime-eventTime, that is, the difference between inTime and eventTime has always been large, which may be related to the data reading mode of Storm and Flink. The blue broken line indicates that the delay of Storm increases as the amount of data increases, while the orange broken line indicates that the delay of Flink decreases as the amount of data increases (the Flink throughput is not measured here, and the Flink delay still increases as it approaches throughput). Even if you only focus on outTime-inTime (that is, the solid part of the figure), you can still find that as QPS grows, the advantage of Flink in latency begins to manifest. 5.9 Windowed Word Count Flink At Least Once and Exactly Once latency comparison

In the picture, × × is 99 lines, orange is the median, dashed lines are At Least Once, and solid lines are Exactly Once. The virtual and real curves of the corresponding colors in the figure basically coincide, so it can be seen that the delay median curve of Flink Exactly Once is basically consistent with that of At Least Once, and there is no significant difference in delay performance. 5.10 Windowed Word Count Storm At Least Once and At Most Once latency comparison

The blue line in the picture is 99, the light blue is the median, the dashed line is At Least Once, and the solid line is At Most Once. When QPS was 4000 or earlier, the dashed lines and solid lines basically coincided; when QPS was 6000, the dotted lines were slightly higher; when QPS approached 8000, it exceeded the throughput of Storm under AtLeast Once semantics, so there were only points on the solid lines. It can be seen that there is no difference in delay between Storm At Most Once and At Least Once when QPS is low, but the difference begins to increase with the increase of QPS, and the delay of At Most Once is lower. 5.11 comparison of throughput between different StateBackends of Windowed Word Count Flink

Flink supports cluster deployment modes of Standalone and on Yarn, as well as three state storage backends (StateBackends) of Memory, FileSystem and RocksDB. Due to the needs of online jobs, the performance differences of the three StateBackends on the two cluster deployment modes are tested. Where the storage path for Standalone is a file directory on JobManager, and the storage path for onYarn is a file directory on HDFS. Comparing the three groups of columns, we can find that there is no significant difference between the throughput of FileSystem and Memory, and the throughput of RocksDB is only about 1/10 of that of the other two. Comparing the two colors, we can find that there is little difference between Standalone and on Yarn. When using FileSystem and Memory, the throughput in on Yarn mode is slightly higher, and when using RocksDB, the throughput in Standalone mode is slightly higher. 5.12 Windowed Word Count Flink comparison of different StateBackends latency

When using FileSystem and Memory as Backends, the latency is basically the same and low. When using RocksDB as Backends, the latency is slightly higher, and because the throughput is low, the delay increases sharply before reaching the throughput bottleneck. Among them, the throughput is lower in onYarn mode, and the delay is higher when approaching throughput. six。 Conclusion and suggestion the performance of the framework itself can be seen from the test results of 5.1,5.5. It can be seen that the throughput of Storm single thread is about 87,000 lines per second, and that of Flink single thread is 350,000 lines per second. The throughput of Flink is about 3-5 times that of Storm. From the test results of 5.2,5.8, it can be seen that the median delay (including Kafka read and write time) when Storm QPS approaches throughput is about 100ms, 99 lines about 700ms, Flink median about 50ms, 99 lines about 300ms. The delay of Flink in full throughput is about half of that of Storm, and with the gradual increase of QPS, the advantage of Flink in delay begins to show.

To sum up, the performance of Flink framework itself is better than that of Storm.

6.2 the weakening of framework differences by complex user logic compared with the test results of 5.1 and 5.3,5.2 and 5.4, it can be found that when the duration of a single Bolt Sleep reaches 1 millisecond, the latency of Flink is still lower than that of Storm, but the advantage of throughput can not be realized. Therefore, the more complex the user logic is, the longer it takes, and the smaller the difference in the framework reflected in the testing of that logic. 6.3 differences in message delivery semantics from the test results of 5.6,5.7,5.9,5.10, we can see that the throughput of Flink Exactly Once is 6.3% lower than that of At Least Once, with little difference in delay; the throughput of Storm At Most Once is 16.8% higher than that of At Least Once, and the delay is slightly lower. Because Storm performs checkpoints on each message ACK,Flink is based on a batch of messages, different implementation principles lead to differences in the cost of At Least Once semantics, thus affecting performance. However, the implementation of Exactly Once semantics by Flink only increases the alignment operation, so it has little impact on the performance of Flink when the operator concurrency is small and there are no slow nodes. The performance in Storm At Most Once semantics is still lower than that in Flink. 6.4 Flink is selected for Flink state storage backend to provide memory, file system and RocksDB StateBackends. Combined with the test results of 5.11,5.12, the comparison of the three is as follows:

6.5 scenarios where Flink is recommended

Based on the above test results, it is recommended to consider using Flink framework for the following real-time computing scenarios:

Scenarios where message delivery semantics are required to be Exactly Once; scenarios with large amount of data requiring high throughput and low latency; scenarios where status management or window statistics are required. seven。 Prospect

There are still some contents in this test that have not been tested in more depth, which need to be supplemented by follow-up tests. For example:

Does the throughput of ExactlyOnce decrease significantly when the concurrency increases? When users spend time to 1ms, the difference in the framework is no longer obvious (the accuracy of Thread.sleep () is only milliseconds), in what range can the advantages of Flink still be reflected? This test only observed the two indicators of throughput and delay, and did not pay attention to the important performance indicators such as reliability and scalability of the system at the statistical level, which need to be supplemented in the future. The throughput of Flink using RocksDBStateBackend is low, which needs to be further explored and optimized. More advanced API of Flink, such as Table API & SQL and CEP, need to be further understood and improved. eight。 Reference content

1. Distributed flow processing framework-functional comparison and performance evaluation.

2.intel-hadoop/HiBench: HiBenchis a big data benchmark suite.

3.Yahoo stream computing engine benchmark test.

4.Extendingthe Yahoo! Streaming Benchmark.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.