Shulou(Shulou.com)06/02 Report--

The little ant said:

"Alipay client Architecture Analysis" series will start with the architecture design scheme of Alipay client, subdivide and disassemble the client in "containerization framework design", "network optimization", "performance startup optimization", "automatic log collection", "RPC component design", "mobile application monitoring, diagnosis, positioning" and other specific implementation. Lead you to further understand the iteration and optimization process of Alipay in the client architecture.

This section will introduce the specific ideas of automated log collection and analysis on Alipay client based on the sharing of "Mobile Analysis solution in Ant Financial Services Group mPaaS" shared by he Xi at OSChina Zhuhai Station. The content will be divided into three parts:

Exploration of the client Analysis Scheme of Alipay

Analysis of MAS Mobile Analysis Framework

MPaaS technical architecture and assistance.

Exploration of the client Analysis Scheme of Alipay

As we have already introduced the architecture evolution and development team size development of Ant Financial Services Group in "opening | discussion of Modularization and decoupling Development in Ant Financial Services Group's mPaaS":

So far, in R & D, Alipay only has nearly 1,000 developers on the Android and iOS clients, the number of lines of code on the client side has exceeded several million lines, and there are nearly 1,000 projects divided by business, and each project has an independent development owner responsible for a specific module. Despite the increasing number of engineers and projects, Alipay is still able to release daily to ensure that the business iterates quickly, while ensuring that the App flicker rate is only 0.01% in an environment with increasingly complex business functions.

So, in such a large number of users and R & D team, how does Alipay ensure the user experience in the process of using it? We mainly measure the client user experience from the following two dimensions:

Static: in the process of application development, pay attention to App's own installation package size, storage, user privacy rights, security policies, etc., to determine whether users are willing to install and use your application.

Dynamic: refers to the actual experience of users, such as App startup speed, flashback, stutter and other stability data, network requests, memory and power flow, after the application is launched.

Launching an application is the most essential operation for users to use any application. From clicking the App icon to displaying it on the home page, the performance of the whole startup process seriously affects the user experience. Alipay client as a super App, startup speed is of course one of the important indicators we pay attention to. The optimization of Alipay in the process of application startup is mainly divided into the following four aspects:

Framework governance:

Sort out the startup process and reconstruct it, and follow the principle of on-demand loading during startup.

Refer to the Pipeline mechanism to specify the timing of business initialization according to business priority.

Develop a unified development specification to minimize the impact of business processes on startup performance.

Business governance:

Load on demand, delay execution.

Thread governance: uniformly manage existing threads and adjust thread priorities.

Icano governance: focus on the main thread Icano, optimize and merge frequent read and write operations, and use unified storage as much as possible.

Technological breakthroughs:

Prevent UI rebrush operations during startup.

Virtual machine optimization, including JIT shutdown, reducing the number of GC.

The basic module is tuned, the time-consuming operation of the main thread is analyzed and optimized.

In addition, the memory, storage, electricity and traffic consumption of App are also important indicators. The specific optimization points are as follows:

Memory:

Memory analysis: memtrace hprof offline memory analysis, traversing objects, marking memory leaks according to life cycle, and determining business attribution based on object creation references.

Native memory: the image library is switched to the native layer, and 4.x bitmap pixel data is put into ashme shared memory, reducing GC.

Memory optimization: object pool reuse, reduce bitmap memory footprint, use smaller diagrams, especially pay attention to three-party H5 pages.

Storage:

Storage analysis: view the application storage size.

Storage optimization: use shared libraries, business-oriented optimization, compressed storage, etc.

Traffic:

Reasons for traffic consumption: analyze various network requests.

Traffic anomaly capture: all network requests in hook. Traffic is aggregated according to host, and the exception is determined when the threshold is exceeded.

Traffic optimization: PC underlying protocol optimization, resource increment downloading on demand, at the same time through the aspect information caller.

Electricity quantity:

Cause of power consumption: monitor CPU utilization, various sensor, gps, weaklock, network connections and other power consumption operations.

Power consumption exception catch: traverse the thread, get the running time of all threads, and compare it with the main thread to determine the exception.

Power consumption optimization: high-performance dump thread stack optimization, through thread mapping caller, evaluation call logic for optimization.

For each of the above optimization points, Alipay has invested a lot of energy in research and practice. For more information on startup performance optimization, please refer to the documents "Alipay client architecture resolution: iOS client startup performance optimization" and "Alipay client architecture resolution: Android client startup speed optimization of garbage collection". For other optimization points, please continue to follow the "client architecture parsing" series of articles.

Based on these contents of optimizing user experience, Alipay constructed a complete set of super App online operation and maintenance system to monitor the abnormal problems occurred in online App in real time, in view of these problems, locate the cause of the problem in the fastest time and find the corresponding solution, and finally repair the online problem in time through dynamic thermal repair technology, and finally form a closed loop of online quality assurance to ensure the stability of application operation.

Analysis of MAS Mobile Analysis Framework

Next, it introduces in detail how to implement the mobile monitoring framework in the super App operation and maintenance system.

Mobile Analytics MAS (Mobile Analysis Service) realizes product core index monitoring, provides basic analysis such as page, device, retention and performance, and supports high-level analysis such as custom event analysis and funnel analysis through the collection and analysis of multi-terminal buried data from mobile client, H5, Mini Program, PC, etc., to help enterprises better complete business monitoring, user insight and behavior analysis, guide product iteration, and refine product operation. Assist marketing decision-making and accelerate business commercialization. It is mainly divided into the following four stages:

The complete link of the whole mobile analysis is viewed from left to right, that is, the client carries out the data burying point by calling the interface of the buried point SDK, and after the buried point SDK formats the log, it first writes the local file on the client side, and after meeting the trigger conditions for log reporting, report the local log to the log server and clean up the local log file to reduce the storage size. After receiving the logs reported by the client, the log server is synchronized to the computing platform, and after offline calculation and real-time calculation, the results are displayed for monitoring, analysis, search, recommendation and so on.

Next, we will introduce the whole link logic of data analysis in detail from the four stages of the mobility analysis framework.

data acquisition

According to the timing of data collection, application scenarios, and the final use, we divide the data collected by the client into the following categories. Combined with the modular development framework of mPaaS, reporting live burial points, background burial points, page automation burial points, performance burial points and H5 burial points, which are automatically collected by the client SDK, there is no need for developers to call the API manually. Developers only need to pay attention to their own business logic and count the logic burial points that need to be monitored.

In order to reduce the impact of frequently reported logs on application performance, the data collected by the client will be saved locally in advance and synchronized to the log server in the following three ways:

Automatic escalation: when certain conditions are met, the client buries SDK to report automatically, including

Each cold start of the program triggers the logic of checking log reporting.

When the program enters the background, it will immediately trigger the report.

When writing a log, a certain type of log is reported when it reaches 40 by default.

Real-time monitoring: more important client logs, such as exception and application flashback logs, can be reported in real time, resulting in a report to facilitate real-time monitoring at the backend.

Dynamic control: on the basis of automatic reporting, the trigger conditions of client log writing and log reporting are modified through the switch values sent by the server. For example, in the case of large traffic concurrency, in order to reduce the pressure on the log server, the client is controlled to write and report only abnormal or flashback logs, ignoring the statistics of behavior logs.

Data calculation

The logs reported to the log server will be synchronized to the computing platform for calculation. The backend mainly includes the following systems:

Mdap: log collection gateway, which is responsible for collecting client buried logs. After receiving the logs, they are directly transferred to the JStorm cluster for calculation.

JStorm: a real-time computing engine that parses logs in real time according to processing rules and stores needed data into the database.

SSDB: kv data storage layer, which uses leveldb at the bottom and supports gigabyte records in a single table.

ZooKeeper: cluster management, inter-component service discovery.

Data application

The results calculated by the computing platform can provide users with data analysis services such as user analysis, event analysis, behavior, performance and so on.

Basic analysis: focus on the general analysis of App, including daily login users, new users, duration of use, user retention, page analysis, access path and other basic analysis.

High-level analysis: for App to focus on specific business analysis requirements, provide a flexible multi-dimensional analysis capability, and provide hot repair reports to help you understand RPC, repair, rollback related information, and so on.

Performance analysis: provide statistical functions of flicker, jam and stutter. When a performance problem occurs on the client, the mobile analysis service provides real-time viewing of performance analysis statistics.

Log management: support real-time search and query log by keywords, or real-time control of client log reporting logic through the server switch.

Data decision

On the basis of the data application in the previous step, it can be combined with big data, marketing platform and push platform. According to the buried point data obtained by mobile analysis, after marking, circling people, user portraits and modeling through big data platform, you can launch a marketing campaign on the marketing platform, specify the type of activity, activity algorithm, participation crowd and event prize, through message push, data synchronization, dynamic release and other forms. Touch the client to achieve the purpose of customer access, activity promotion and operation guidance. At the same time, combined with the scene requirements of operational activities, a complete digital operation system is formed to monitor the number of participants, activity distribution rate, write-off rate and so on, to observe the effectiveness of the activity.

Technical Architecture and help of mPaaS

The technical accumulation and architecture practice of the mobile analysis solution in Alipay introduced above has been opened to the outside world as part of Ant Financial Services Group's financial technology through the mPaaS mobile development platform. MPaaS (Mobile Platform As A Service), a mobile development platform derived from Alipay App, provides a cloud-to-end one-stop solution for mobile development, testing, operation and maintenance, which can effectively lower the technical threshold, reduce R & D costs, improve development efficiency, and help enterprises quickly build stable and high-quality mobile App.

On the mPaaS mobile open platform, we output the local log, burial point, automatic burial point, performance monitoring, Crash report, diagnostic log and other modules in the mobile analysis framework as independent components. Any App can add corresponding components through the mPaaS plug-in and integrate these functions in the current application. With such a simple operation, your application can have the same powerful mobile analysis and monitoring capability as Alipay.

After the client integrates these components related to mobile analysis, users will generate corresponding logs in the process of using APP. After data collection, data reporting, data calculation and other processing, the calculation results will be synchronized to the market of mPaaS mobile analysis to display, including the basic application overview, performance stability data, traffic trend and so on. It is convenient for developers to monitor the general situation and stability of APP in real time. Discover online problems in real time and fix them.

At present, mPaaS mobile development platform has served many enterprises, including Ant Financial Services Group's internal Hong Kong Alipay, Internet Merchant Bank, word-of-mouth merchants, etc., as well as a large number of external ant ecological partners, including 12306, Shanghai Metro, Guangzhou Metro, Guangfa Bank and so on. Adhering to the concept of "bringing small and beautiful changes to the world", we have reconstructed the client of national-level App such as 12306 through mPaaS, so that people can use the App of a good experience to buy travel tickets, and it only takes 2-3 months to build a 12306 with a mature framework like mPaaS.

In addition to the Hong Kong version of Alipay in 12306, Guangfa Bank mobile banking and discovery of a number of clients, also completed the business restructuring in just a few months.

ATEC City Summit Shanghai

On January 4, 2019, a journey to explore the forefront of financial technology-Ant Financial Services Group ATEC Science and Technology Conference is about to set sail, are you ready? The little ant has prepared full of strategy benefits for everyone, waiting for you to come and get them! Learn more about Ant Financial Services Group ATEC Science and Technology Conference. Remember to keep following Little Ant (official WeChat: Ant Financial Services Group Science and Technology) ~ ~

Ant Financial Services Group Financial Technology official website:

Https://tech.antfin.com/articles/activity/atecshanghai

ATEC Science and Technology Conference:

Ant Financial Services Group ATEC (Ant Technology Exploration Conference) Science and Technology Conference is the largest technology event held by Ant Financial Services Group in China, which aims to share the development trend and landing practice of new technologies to partners and technical professionals all over the world, and to bring equal opportunities to the world through the exploration and discussion of advanced cutting-edge technologies. The ATEC conference is on its way. In the past year, Ant Financial Services Group ATEC Technology Conference has traveled through the global financial technology center cities such as Hangzhou, Silicon Valley, Singapore and London. After that, he will visit various financial technology center cities in China to share Ant Financial Services Group's insight into the forefront of financial technology with the local audience.

Application method for ATEC Science and Technology Conference & Welfare:

The tickets for this conference adopt the audit system. Guests fill in their personal information to register, and within 3 days after registration, they receive a text message indicating that the registration is successful. The deadline for registration is 24:00 on December 31, 2018.

The first 50 applicants will be approved on a first-come-first-served basis.

The little ant has also prepared the exclusive welfare invitation code for the readers of this account: SF2B3A

What are you waiting for? click on the registration link below or read the original text or scan the QR code. The little ant is looking forward to your arrival.

ATEC registration link:

Https://alipaytech.mikecrm.com/l6YPW6V

ATEC registration QR code:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.