Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the method of Flink real-time computing big pressure test". In daily operation, I believe that many people have doubts about what is the method of Flink real-time calculation of big pressure test. The editor consulted all kinds of data and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "what is the method of Flink real-time calculation and pressure test". Next, please follow the editor to study!

1. Background:

Approaching the double 11 promotion, in order to avoid a high peak flow, e-commerce companies will pressure test the system. Has been doing real-time data calculation, the application is based on Flink, receive kafka messages, data statistics, including: pv, uv, dau, unit volume, turnover and so on.

In order to ensure that there are no problems in the application during the great promotion period, it is necessary to carry out a real-time calculation program for pressure testing. Because the statistical data are divided into two types (flow data and order data), the two kinds of data are tested in different ways.

Second, pressure test preparation

(1) selection of stress test time: generally, it is chosen in the early morning, which has the least impact on the business.

(2) Cluster preparation: set up master and standby clusters, and set up two sets of kafka cluster, Flink cluster and redis cluster respectively, when resources are abundant. (the active and standby clusters can not only achieve disaster recovery and high availability, but also easily achieve stress testing in the process of stress testing.)

(3) if the resources are limited, the main and standby clusters cannot be built, and the pressure test can be carried out, but there is no guarantee in terms of disaster preparedness and high availability.

III. Pressure testing practice

Case 1: when there is an active and standby cluster (the stress test is relatively simple)

(1) notify the downstream to switch the queried data to another set of clusters (clusters without stress testing)

(2) adjust the kafka message site and adjust the news squeeze to about several times of the peak of last year's big boost.

(3) restart the Flink application

(4) View monitoring: kafka monitoring, Flink monitoring

Kafka message squeeze monitoring

Flink data processing monitoring

Case 2: if there is no master / slave cluster (since there is no master / slave cluster, the stress test may have an impact on the downstream statistics, so you need to notify the downstream)

1. Pressure measurement of flow data (due to the large amount of data, it is difficult to do idempotent processing, which has a great impact on the downstream)

(1) notify the downstream to arrange the pressure test to be carried out in the early morning.

(2) stop the Flink application, resulting in message squeezing, reaching the number of extruded messages.

(3) start the Flink application, start consumption, and view the monitoring metrics

Note: instead of resetting kafka message sites, the number of messages is held, which only makes the downstream data indicators unavailable for a short time, and the overall impact is not significant.

two。 Order data pressure test

Without idempotent treatment, the pressure measurement can also be carried out by using the method of flow rate.

Have idempotent treatment

(1) notify the downstream to carry out pressure test

(2) reset the kafka message site

(3) restart the Flink application, start consumption, and view the monitoring metrics

Note: due to idempotent processing, reset sites will not affect the downstream statistics.

At this point, the study on "what is the method of Flink real-time computing big pressure test" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.