Shulou(Shulou.com)06/03 Report--

In 2007, the number of people offering red envelopes for Alipay Wufu red packets was 167966715 (about 168 million); the number of participants on that day was 220 million; at the peak of business, the peak of loose-leaf main page was 81W 90W/s; the peak of sweeping fortune was 22W picks; the peak login of New Year's Eve on that day was 29W picks; and the peak of New Year's Eve's lottery was 90W/s. In this article, Ant Financial Services Group technical expert Skyglass Fei went deep behind the technology and explained to you the actual combat experience in the aspects of overall activity stability guarantee, capital loss prevention and control, operation and maintenance deployment, production guarantee, monitoring management, security protection, and so on.

Business model

This year's Alipay Spring Festival red packet mainly adopts two modes: AR red packet and Wufu red packet. AR red packet is the embodiment of innovative business, which is mainly divided into three steps: hiding, finding and opening. The hiding is to scan the specified object at the current position, enter the amount and number of people, and after the payment is completed, the red packet appears on the map; the user can find the red packet on the map or find the red packet by scanning the specified object. after finding the red packet, the user can use the clue map to open the red packet. Another way to play AR red packets is merchants' red envelopes, which are mainly used for merchants' promotion. Merchants can hide their Logo in the geographical location of all stores, by guiding users to scan Logo and open red envelopes in designated stores, thus playing the role of merchants' promotion.

Wufu red envelopes, also known as professional blessings, are similar to those of last year: collect Foca, exchange Foca, and wait for the prize after collecting Foca; the difference between the two is that in the collection of Foca, this year's forms are more diversified: sweeping Fu, ant forest watering and diamond members can get omnipotent; at the same time, the amount of money is allocated in a random way to ensure that everyone has a share.

Business core indicators

The figure above shows the core indicators of this year's Alipay red packet business: the number of lottery winners for Wufu red packets is 168 million; the number of participants on that day is 220 million; at the business peak, the peak value of the main page of the event is 81wamp s; the peak value of sweeping fortune is 22wamp s; the peak login of New Year's Eve on that day is 29wswap s; and the peak number of New Year's Eve lottery is 90w/s.

So how does Alipay support such a high concurrency business and such a large crowd? What is the technical system behind it? Let's take a look.

Alipay support technology system

The technical framework supporting the realization of Alipay red packet products can be summarized into six elements: stability guarantee, capital loss prevention and control is the core; operation and maintenance deployment, production security is the cornerstone; monitoring and management, security protection is an important part. These six elements support the product to achieve the overall operation of the technical architecture and finally come to a perfect end.

The technical architecture of Alipay red packet products is shown in the figure above. Based on the requirements of business variability and user experience, H5 and Native are mainly used on the terminal: H5 is mainly used for changeable services such as ant forest, Foka task and Foka ranking; Foka home page, AR map, AR scan, AR engine, AR red packet and other services adopt the way of local Native, because its user experience is very good. Of course, the overall client architecture is based on the general capabilities of Alipay client-H5 container, such as framework, communication, network, login, security and so on.

On the server side, both Foka business platform and AR business platform are based on Alipay's existing financial capabilities, in which Foka business platform is based on the existing marketing award fund processing capabilities, in the process, the use of recommendation, voucher center, ant forest product capabilities, the most important thing in Foka business is to configure the background (this is because it is a configuration product) So we set up the Spring Festival configuration background and marketing configuration background. The AR business platform is based on the existing financial capacity of individual red packets and merchant red packets, in which image recognition services and geographic location services are the two most important points in this business. To sum up, the server side depends on the sharing capabilities of Alipay's new financial engine, such as monitoring, social networking, membership, accounting, payment, security, SYNC, middleware and so on.

In the aspect of storage, in addition to the general MySQL storage, a large number of relational database Geabase is introduced, tair is used for distributed cache, and OSS is used for picture storage.

Let's explain the six elements that support product implementation one by one.

Operation and maintenance deployment

The problem to be solved in the deployment of operation and maintenance is the resource gap. Under normal circumstances, the number of machines in East China 1 and South China computer rooms can not meet the needs of Spring Festival activities, so they need to move around. In the process of solving the resource gap, elastic resources are used to save costs and "bounce" the key links to the cloud; after the event, the traffic is moved back from the cloud to the offline computer room. Normally, East China No. 1 computer room and South China computer room bear 40% and 60% of the traffic respectively, and they are non-cloud machines. In the Spring Festival red packet business, Alipay cuts 60% of the traffic to the East China 2 server room and puts it on the cloud, with the help of Aliyun's powerful server capability. in addition, 15% of the cloud machines will be deployed in the South China server room, that is to say, 75% of the machines in the Spring Festival red packet business run on the cloud. After the event, the traffic will be cut out, resulting in great cost savings.

Production guarantee

Production guarantee is another cornerstone of the technical architecture. During the launch of the business this year, code development and product launch accounted for only 20% of the total effort, and 75% of the effort was devoted to drills and stress tests. For the drill and stress testing, Alipay designed three sets of environments, namely, the pressure testing environment, the exercise environment and the formal environment. Terminal, server and storage all support these three environments to switch back and forth, and the three environments are based on whitelist isolation control mechanism to prevent the risk of online stability caused by crosstalk among the three environments.

Optimize the product process in the drill and polish the code performance in the pressure test. When the product is launched this year, the business process and the final formal environment of the product business process has changed greatly, and the level of the code has also changed a lot. Using these three sets of environments, through pressure testing to optimize the code, through online drills, constantly improve the code, products, and finally achieve the basic stability of online product status. What is worth mentioning here is grayscale zipper. In the lottery section, the grayscale lottery function is designed this year, that is, one day in advance, some users can open the lottery in advance and fix a serious configuration problem in the process of lottery opening in advance. To avoid failures when officially launched, so grayscale zipper can be said to be the most critical link in the production guarantee.

Monitoring and management

Monitoring and management and security protection are two important components.

Monitoring is the "eye" of the product, which provides a basis for technical and product personnel to find online problems and make business decisions. This year, according to different monitoring scenarios and requirements, Alipay has deployed four sets of monitoring modes: the first set of monitoring mode is to use Xflush to analyze the stability log of the system, resulting in system monitoring, business link market and SLA current-limiting market. This part is mainly to provide online operation status monitoring for various technical leaders. The second set of monitoring mode is to use the Kepler real-time computing platform to analyze the business log, produce some business indicators, and finally produce a large TV screen, a mobile live studio and a marketing market. The third set of monitoring mode is to use the Hainer real-time computing platform to analyze the client log, resulting in resource download market, performance stability market, basic network market, which can effectively monitor and early-warn the running status of the client; the last set of monitoring mode is for customer service security log analysis, resulting in a large security screen to monitor user complaints.

Safety protection

Only by considering safety protection at the beginning of product design, can we take precautions against problems in the course of product operation. This year, Alipay adopted three security strategies on AR red packets and Wufu red packets. The first two lines of protection are mainly aimed at the clue diagram and LBS of AR red packets, and the first protection is against the clue graph of AR red packets. Through the clue graph occlusion algorithm, a balance can be found between the clue can be identified and the clue is not disclosed. The second security protection is aimed at the monitoring of LBS tampering. The strategy adopted is that the terminal collects the user's LBS, displacement and other data, and then reports them to the server. When the user receives the AR red packet, the server analyzes whether the LBS, displacement and other data are drifting, and intercepts the abnormal situation. The third protection is based on big data's security policy protection. Big data's analysis of users' data is carried out by the Alipay security team. Based on big data's analysis, corresponding security policies will be produced. These security policies will protect the key processes of the product, mainly winning Foca, exchanging Foca, opening Foca, receiving AR red packets and sending AR red envelopes. Different security protection measures are adopted for different links. To ensure the effectiveness of user operations.

Stability guarantee

In the stability guarantee, the first prominent point is to scatter the picture recognition in the terminal, which is because the image recognition algorithm is very performance-consuming. Picture recognition is faced with two choices: one is to complete picture recognition on each terminal and communicate directly with the user, which has a better experience, in addition, it also relieves the pressure on the server side, and because the picture does not have to be uploaded to the server side, it can save the cost of traffic, but the biggest problem it faces is insecurity, because the user can tamper with the picture recognition result at the terminal, thus deceiving the server side. Have a higher risk Another option is to carry out picture recognition on the server side, that is, the picture is uploaded to the server side, and the recognition is carried out through the server side algorithm to ensure the security of the recognition results. its disadvantages are poor user experience, high consumption of server resources, and consumption of a lot of user traffic.

Therefore, you need to find a balance between these two methods. Alipay uses different image recognition methods for different image recognition scenarios. For example, getting a red packet on a map uses a strict match on the client, and the server verifies it again according to the policy, with security first; scan the scene of receiving the red packet, using Top n match on the server side, and check the server signature when receiving it, also for security considerations. The actual peak values of these two scenarios are not high, and putting them on the server side is within an acceptable range. The other three scenarios, including Coca-Cola red packet (red packet / Foca), word-of-mouth activity (red packet), and any Fu activity (Foca) are mainly to identify Fu word and merchant Logo, in which privacy issues are not involved, so pure client-side matching is used for merchant Logo, which is no longer recognized by the server, which avoids the resource consumption of the server. For any blessing activity, the client strictly matches the Fu word, and when the client cannot recognize the Fu word, it is identified by the server side. This method can save more than 70% of the resource consumption on the server side.

In the stability guarantee, the key link multi-gear switching is realized for the scan link. The key points of the scan link mainly include terminal blessing word recognition, nearby red packet clue matching, service end blessing word recognition and Foca collection. Alipay has designed three gears for scanning links: gear one, complete mode, that is to recognize Fu characters and red packets. This mode can withstand the peak value of 12W/s. In this mode, the system performance can be adjusted by adjusting the number of matching red packets nearby. The second gear is to remove the red packet Top n matching link, and the server side only retains the two functions of Fu word recognition and Foca pick-up, which can withstand the peak value of 50W/s. In this mode, the system performance can be adjusted by adjusting the complexity of the server-side Fu word recognition algorithm; third, only Foca pick-up function is retained on the server side, which is expected to withstand the peak 60W/s, saving resource consumption such as picture download.

In the process of practical use, gear 2 has met the requirements, but the idea of multi-gear provides a good support for the guarantee of stability.

There are many details of stability guarantee, and each detail is very important. In addition to the above two points, Alipay also considered the carding and avoidance of some risk points as a whole, formulated operation manuals before, during and after activities, and formulated corresponding emergency handling mechanisms. and carry on many simulations to the key business process.

On the terminal, stability operations such as current-limiting non-perception, resource pre-download, user operation data cache, discrete lottery time, dynamic configuration of data items and switches are adopted; on the server side, full-link carding, full-link voltage testing, current-limiting protection, emergency fuse mechanism and so on.

Capital loss prevention and control

Capital loss prevention and control is another core point in this year's red packet activity, because the amount of money involved in AR red packet and Wufu red packet is huge. In addition, this year's prize pool allocation strategy adopts a random allocation method, which further increases the difficulty:

It is difficult to calculate. The lottery is only 18 minutes. How can 200 million be randomly allocated and distributed within 18 minutes? It is difficult to check and the risk of capital loss is high. it is not just to calculate the amount, but to ensure that the sum and total amount of each person is the same. It is difficult to distribute the prize pool, and all the 200 million cash will be handed out cleanly, which involves forecasting, bottom guarantee and other issues.

In view of the difficulty in calculation, a multi-gear random mechanism is introduced, and multiple lottery stalls are designed to look like random results instead of a purely random way; in addition, the way of calculating prizes in advance is adopted to calculate the bonuses of some users in advance. When actually opening the lottery, only check the original calculated amount. Based on these two strategies, Alipay designed a new model for capital loss prevention and control this year, which includes three links: lottery calculation, lottery draw and prize distribution, which is ensured by monitoring, early warning and checking mechanism. among them, the prize calculation link is designed to be repeatable, and when an anomaly is found, the bonus can be recalculated, the lottery link supports rapid correction, and the prize allocation link supports rapid account transfer.

The checking mechanism is the most critical link in the prevention and control of asset loss, which is based on the real-time computing platform this year. The check mechanism mainly includes two check strategies: one is real-time check, which uses real-time check for the key changes of capital flow and the switch of capital chain; the other is asynchronous check, that is, the so-called 15-minute check, hour check and Terror 1 check. Asynchronous check is used for all the links related to funds to ensure the final consistency.

Real-time verification adopts three basic strategies: the first strategy uses Qiankun mirror technology to check the input and output parameters of the interface in real time; the second strategy uses scheduled tasks to ensure quasi-real-time performance as far as possible. The third strategy is to mix the checked code fragments into the business code to ensure the consistency of real-time verification. Asynchronous check, including 15-minute check, hour check and Tester 1 check. The technical framework adopted is mainly the combination of DRC (remote multi-active data synchronization) and TT. The data of DB and log are stored on ODPS according to 15 minutes, hours and days, and then the overall consistency is ensured by the corresponding checking mechanism.

In the process of calculating the lottery, drawing the lottery and handing out the prize, before the next link begins, the accuracy of the previous link must be ensured by checking the strategy, including the total amount and details; in this way, you can make a smooth transition to the next stage without having to worry about problems in the previous stage.

Summary

Alipay Spring Festival red packet products take operation and maintenance deployment and production guarantee as the cornerstone, stability guarantee and capital loss prevention and control as the core, supplemented by two important components of monitoring and management and security protection, which perfectly solve the two problems of high concurrency business and huge personnel participation of Wufu red packet and AR red packet. A series of indicators such as the peak value of the main page of 81w/s, the peak of fortune sweeping of 22w/s, the peak login of New Year's Eve of 29w/s, and the peak of New Year's Eve lottery of 90w/s have been achieved.

Original text from: Yunqi Community; original link: https://yq.aliyun.com/articles/71053

Click to read more and see more details

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.