Shulou(Shulou.com)06/03 Report--

The little ant said:

During the decade of Singles' Day, the exponential growth of transaction size constantly challenged people's imagination, and for the ant technology team, it was not only a consumption feast, but also countless experiences close to the limits of stress and anxiety, but also a training ground for technology. Now for Ant Financial Services Group, Singles Day is no longer limited to a technology project, but more like a social project, which can be called "coherent, social technology collaboration".

Picture | Oriental IC * CYZONE close-up, disconsolate stories of the Great Times

Isn't the Alipay team just like the red-lacquered sculpture?

On the one hand, they are in awe and humility of technology, while on the other hand, they have to run with their lives.

"Singles' Day" is right now, but the atmosphere in Ant Financial Services Group's new park is clear and the crowd is not in such a hurry as in previous years.

When entering the park, the taxi driver held the steering wheel with his left hand and said with his right hand that autumn is the best season in Hangzhou, and of course, spring is not bad. The sun shines violently on the buildings of the park, making the golden chestnut glass deep and clear.

There are still many things in the new park that are not well known.

Every 3 minutes, one person will twist out cigarette butts and squeak on the garbage can on the left-hand side of the door of Building 2; the girl in the visitors' hall uses the fat sea to brew 4 pots of tea with ginseng slices every day, 12 glasses are turned upside down, and the cup handle is 30 degrees to the right; the cleaners in orange coats in the park constantly sweep the fallen leaves. They work eight hours a day in three shifts, always preparing three spray cans and one insulation cup on the cart. In each flower bed, you can usually use bamboo clips to reach three cigarette butts or pieces of paper.

The temperature difference between day and night in autumn here is only 6 degrees Celsius, but some people are thinly dressed sooner or later. When a takeout in a stormsuit calls on the cell phone, the microphone is 20 centimeters away from his mouth, with an average voice of 70 decibels. In the afternoon, many restaurant employees squat at the door to smoke, only Starbucks continues to flow. The big cake and mini cake here are scheduled for three days, and the cashier occasionally marks Mr. Dong with an ink pen. The staff restaurant serves meals four times a day, and an additional 18 restaurants are arranged between the buildings.

There is a free gym of more than 1000 square meters in the building, and the price of private training is only half that of the outside, and the girl in Nike running shoes brings her corgis to use two treadmills at the same time every day, even though she has nothing to pick on.

When you talk to the fourth person in the sunken square, the security guard in the jacket will stare and ask your identity before you get hairy. Oh, it's a reporter, never mind, do your job.

This is Ant Financial Services Group's new headquarters, which costs more than 1.1 billion yuan and covers an area of 180000 square meters, with complete facilities and functions, comparable to small towns. It is the work of NBBJ architecture firm, also known as "Ant Z Space".

The powerful complex building accommodates Hangzhou people and New Hangzhou people here, watching their every business and innovation, carrying the wealth and dreams of tens of thousands of people, as well as recording the pride and anxiety of each individual.

During the double 11 decade, the exponential growth of transaction size constantly challenged people's imagination, while behind the rapid expansion, there were countless experiences on the verge of stress and anxiety for the technical team.

Imagination and anxiety initially formed a net for Ant Financial Services Group's technical team, and then woven into a fine and thick cocoon shell. In the three or four years since 2010, people will always feel some stutter and discomfort in front of the consumption of Singles' Day, while inside it is the game and even struggle between this group of engineers and desire and imagination, and constantly break the shackles at many moments approaching the limit of anxiety.

In August 2017, Ant Financial Services Group officially opened the new park called "Z Zone". Image worm creativity

"for dozens of seconds, is it worth it? "

On the morning of autumn in Hangzhou, it was very cold. Huang Yong got up early and went to Lingyin Temple with several colleagues. This thousand-year-old temple in the mountains, has always been exuberant incense. In recent years, temples have become fashionable and environmentally friendly, sending out fragrance free of charge. He invited him to order and say good-bye. During the interview, I asked which Bodhisattva I worshiped. Huang Yong frowned and was happy. "I really don't know him."

The heart of burning incense is sincere, and few people who come to make a wish are bigger than his wish. as the PM (project manager) of this year's Singles' Day payment guarantee, he has to manage this project involving hundreds of millions of people in every detail.

Every Singles Day, Ant Financial Services Group's project team members always provide for the second Master Guan, put on red underwear, change into red war robes, save a few bottles of red wine and burn a few burners of incense. According to the words of Chen Liang, head of the Alipay double 11 support team (nickname Junyi, researcher of the Technical risk Department), this is an awe of technology.

But in fact, it is not just technology that should be feared. As an unprecedentedly complicated project, Singles' Day has countless random possibilities in the details of everything, which has long been beyond the boundaries of human control. What Huang Yong can do is to develop a "disaster recovery" mechanism and try his best to approach the impossible "certainty". For example, on the day of the interview, Huang Yong just e-mailed all the members who were going to enter the Guangding (Alipay double 11 War Room) on the night of November 10, carefully explaining the theme "what if the teacup was knocked over on the computer that night?"

In 2012, the PM colleague in charge of Alipay's double 11 project invited back a shadow Guan statue from Xi'an. Everyone thought it was fresh and offered cigarettes, yogurt and fruit. Since then, whenever an important project has been launched, someone has always mentioned to the company to invite second Master Guan. When Entrepreneurship visited Ant Financial Services Group this time, there was a bronze statue of the second master in the war room, and the confession should have been placed a long time ago. The second master seems to have begun to bring good luck. The classmate who invited the bronze statue won a big prize in the company on Singles' Day the year before last.

When Ma Yun took several partners to visit Xixi Park and boarded the Guangding (Alipay double 11 War Room) on Singles Day one year, a female investor was surprised to ask, "are you engineers fashionable to worship Guan Gong?" Laugh as soon as you are handsome and righteous, but still say it in awe.

Whether it is faith or awe, Singles' Day is obviously worth it. In a decade, from the initially almost imperceptible promotions, driven by a mixture of desire, emotion, responsibility and creativity, it has grown into a behemoth that continues to break through the limits of imagination.

In 2009, the daily turnover of the first double 11 Shopping Festival was more than 50 million yuan. In comparison, the daily transaction volume of Alipay topped 1.2 billion yuan that year. "I remember there were dozens of brands involved, and the feeling about it at that time was that Taobao did an activity," recalled Ni Xingjun, president of Alipay business group (alias Miao Renfeng). But he did not expect, no one expected, from the following year, Singles' Day began to refresh everyone's imagination, and now looking back at the growth curve, it maintained a numerical growth rate in some years. The line is really steep, but think about it, at that moment, the unknown and growth bring people a stronger sense of psychological impact.

In the memory of Ant Financial Services Group CTO Cheng Li (nickname Lu Su), double 11 in the years after 2010 was a huge test for Alipay's technical team like the movie "2012". "if you put a boat there, there is a big wave on it, and no one knows if you can hold it. If you can't carry it, it's gone." "

The ship can only be built in advance according to the existing imagination, but when dealing with huge waves, random loopholes must be temporarily remedied. Random means uncertainty, and huge randomness and uncertainty further exert greater pressure on the team. Cheng Li remembers that Li Jin, the current vice president of Aliyun, was in charge of the double 11 project at Alibaba Group at that time. "when he couldn't stand it, Li Jin had to drive to Longjing Mountain and open the window to sleep overnight. He said he was under too much pressure and needed oxygen. "

In 2010, Alipay, which welcomed double 11 for the second time, experienced a "4-second shock" that was later widely known. At 23:59:30 on the 11th, half a minute before the end of double 11, Alipay's core accounting system suddenly gave an alarm and resources were about to be exhausted. At that time, the entire Alipay accounting database has not been split, once the system crashes, all business will be dead, Taobao and Alipay will cause catastrophic losses. When the engineer shuts down the application of an accounting system and releases resources, there are only four seconds left before the database crashes.

Technology alone was an account that could never be calculated at that time.

Before Singles Day in 2012, the Alipay technical team had done all the conceivable stress tests, but something went wrong during the peak that night. Gong Jie (nickname Yuan Yue), an operation and maintenance engineer, remembered that the threshold set by a data channel in the background was too low at that time. It caused a temporary outage, but the system determined that it could not respond, so it was automatically removed, and then the servers hung up one after another, "like an avalanche." The system did not resume normal trading until it was downgraded and cut off part of the traffic. According to Cheng Li, the safety tungsten wire was fused by high-frequency trading and temporarily put up a copper wire to cope with it.

At this time, too large and complex system, manpower has been unable to complete a comprehensive and effective test. Gong Jie said that because the database could not bear the pressure in the past two years, a large number of stress tests had been done at the application and DBA level in 2012, but in the end, it was precisely the "intersection" that had not yet been pressed.

In the interview, Junyi said with a wry smile that at that time, the Singles' Day was full of confidence and lived in fear every year. In the years when Singles Day was the most stressful, the entire Alipay technical team spent several months or even half a year on "military training", doing various technical structural adjustments and system tests. Junyi initially had doubts that most of the time and energy spent by the whole team was only to contribute to the seconds at the peak of Singles' Day.

"does it have to be like this? "

"is it worth it? "

But time will give ultimate meaning to all the previously unknown things, and Singles' Day is the product of such an era in which the meaning is gradually extended. "at that time, Taobao was our biggest customer, and we had to serve well," Junyi said. According to Jack Ma in his early years, apart from the customer relationship, Taobao Tmall and Alipay are more like husband and wife, and it is precisely under the pressure of the business of Taobao Tmall that the technical ability of the Alipay team has been unprecedentedly stimulated. An engineer who joined this year makes no secret that the core attraction of his appointment to Ant Financial Services Group is Singles' Day. "for engineers, there is no more challenging project than double 11. "

Gong Jie also realized later that the peak of tens of thousands of transactions per second achieved by a credit card team in the laboratory environment had long been left behind by Alipay in actual combat. On Singles Day in 2017, Alipay's transactions peaked at 256000 transactions per second.

According to Li Zheng, a senior technical expert (nicknamed Mi Heng), the technical team has carefully divided the 48-hour workload of Singles' Day in recent years. "We have made a very detailed operational manual, which has a lot of steps." according to different time points, you have to implement. "

In addition to technology, Singles 11 is a huge collaboration system involving different departments, different teams and different enterprises in a wider range. Chen Liang, vice president of Ant Financial Services Group Group (nickname Guansheng, head of brand and public communication department) remembers that around 10:00 on the night of Singles Day in one year, a million order numbers in the trading system of a large state-owned bank glowed, and subsequent orders could not be generated, so all payment orders from the bank could not be executed in the last two hours of the night. "there will always be problems that you can't imagine. We have made all the preparations, and we have no choice but to come and store the water. "

Think about it, it's like a rocket into the air, Ni marched on the table and said. How many software and hardware technical links, how many parts are assembled and disassembled, in the process of design and manufacture, we can only exhaust all the possibilities that can be reached by the human brain to do the test, but at the moment of ignition, what is waiting for it is a complete completion or an explosion in situ. You just have to wait.

Ni marching felt that whether it was the technician's plan of worshiping Guan Gong, burning incense or the public relations team, it all proved that Ant Financial Services Group's team was in awe of Singles Day. In May 2013, Alipay took off the last IBM minicomputer, then gradually replaced Oracle with a self-developed OceanBase database, and completed the de-IOE project. Now for Ant Financial Services Group, Singles' Day is no longer limited to a technical project, but more like a social project. Cheng Li said that if a clear organizational concept is defined for it, it can be called "coherent, social technological collaboration."

Lu Su (Cheng Li) in the double 11 battle room | A photo provided by the interviewee

He ran in awe and ran.

The buildings in the ant Z space maintain a strange geometric shape, like a Z word with a "pouted ass" and a cement giant wriggling and dancing. But contrary to the bizarre architectural design and messy flow of people on the outside, there are gates and document machines inside the building, building a solid order and access process. Outside, the huge red humanoid sculpture bows to the entrance of the flow of people, humble, but unstoppable.

Isn't the Alipay team just like that sculpture? On the one hand, they are in awe and humility of technology, while on the other hand, they have to run with their lives. During this decade, in addition to Singles' Day, they also have a lot of anxiety to dispel.

Asked about the most unforgettable moment in Alipay's more than ten years of work, the first choice for Ni Marching and Chen Liang was that annual meeting. January 21, 2010, Alipay annual meeting, there is not much internal from the conscious sense of crisis. With its leading market share and glowing business data, everything seems to be going well.

But as soon as the annual meeting began, people found the atmosphere a little strange. Accusations, complaints, helplessness and criticism were first heard from the loudspeakers at the venue, which came from customer complaints recorded by customer service calls.

However, the development of the situation at the scene is not just "reflection". When Chen Liang arrived at the meeting, he received the news that Ma Yun and other senior executives of the Organization Department of Ali Group would be present. Subsequently, the representative of the customer satisfaction Center took the stage to express "how bad our experience is and how users suffer", while the BD team pointed out "how partners have high expectations for Alipay and how disappointed and helpless they are at the same time."

Jack Ma lost his temper at the scene. It sucks, it sucks, it sucks to the extreme.

Chen Liang remembers that it was the only time in years that he had seen Jack Ma lose his temper in public. Jack Ma bluntly pointed out that Alipay is too conservative on many issues, if you do not pay attention to the user experience, "will die slowly."

This is obviously far from the conclusion of Alipay's team's self-evaluation. in fact, at that time, if you compare horizontally, Alipay's product design and market share performance is by no means poor. The team even defined 2009 as the "year of user experience". But looking back, the product experience on the PC side at that time was really not ideal, and every payment needed to solve a lot of problems such as controls, plug-ins and external U shields.

Wang Jian, then the CTO of Alibaba, also gave a very stern comment: "entertain yourself." This even made Ni marching a little confused at the moment. He remembers that some time after the annual meeting, he fell into serious self-doubt. "after so many years of technology, how has it become self-amusement?" Is there something wrong with our understanding of technology? "

Later, he realized that the error appeared in the conversation from the technology to the product, to the business, to the customer. Mission and vision alone are not enough to drive a customer experience. There is a gap between how he originally thought it should work and the realistic expectations of users.

The entire payment industry in China can be divided into three stages according to the evolution of payment methods: from online bank payment to fast payment from 2009 to 2013; the rise of mobile payment from 2014 to 2016; and fingerprint and facial scan payment becoming mainstream from 2017 to 2018.

Looking back now, the annual meeting is a crucial node for the whole Ant Financial Services Group company. driven by this transformation, Alipay has stepped into the era of fast payment from online banking payment. "Sheng Sheng was forced out," Junyi recalls. "if there had been no fast payment at that time, the whole process of China's mobile Internet would have lagged behind by at least two or three years."

Before WeChat Pay joined, Alipay had to debug itself for ten years, looking for development coordinates. When the former entered the game, the Alipay team responded: wow! We have competition.

"We have never encountered competitors like this. Competition is normal, but the outcome depends on the energy of competitors. WeChat Pay is a very respectable competitor. "that's what Chen Liang said.

The emergence of WeChat Pay prompted Ant Financial Services Group to promote the promotion of ideology once again. Today, the wind is light, but at that time the wind and clouds are surging, and the mood is depressed in every way. Back on January 26, 2014, Tencent launched WeChat red packet, who immediately became active in the WeChat group as a virus and broke out on New year's Eve. Data show that from the same day to the eighth day of junior high school, New Year's Eve, more than 8 million users participated in red packet activities, more than 40 million red packets were received.

In sharp contrast to the popularity of WeChat red packet, the response to Alipay's "lottery head" is mediocre. The latter was launched on the 23rd, three days early.

"one red packet for Wechat exceeds what Alipay has done for eight years. This sentence quickly spread, and Jack Ma later used the "sneak attack on Pearl Harbor" to evaluate Tencent's launch of WeChat red packet.

Chen Liang has a particularly deep memory of this incident, and he participated in the product discussion of Alipay red packets. Because he had also worked in Guangdong and knew that there was a local custom of begging for red envelopes, he gave the suggestion of "begging for red envelopes". But what Wechat did was to "give out red envelopes," Chen Liang recalled. Some people seemed to have raised this point during the discussion at that time, but the product design did not adopt it in the end.

In fact, even if Alipay adopted the design of handing out red envelopes at that time, it might not have a chance to win at that time-there was no relationship chain, no community, and no overall preparation from the trading system to the account system. But Chen Liang still felt remorse, uncontrollable remorse, and even blamed himself for not being as skilled as others.

Seeing the rain of red envelopes and praise in the media group, Chen Liang did not want to go to Wechat. "I don't want to talk anymore. I don't dare to talk anymore."

He wanted to go to his friends to find a solution and to refute the saying that a red packet lasted for eight years, but as soon as he opened his mouth, he was silent, and the market reaction spoke for itself. But he still thought over and over in his heart, how come we didn't think of other people's idea, why didn't we?

But it's a thing of the past. Although company-level anxiety persisted into 2016, Chen Liang has learned to keep anxiety out of his life. It has been proved that anxiety is useless-the anxiety of the previous two years did not make any sense except to make himself nervous and deformed.

In fact, accepting competition of this magnitude may be accepting the gift of fate in a sense. Chen Liang was always thought by his young colleagues that he was insensitive to difficult things, and he thought the reason was that he had never had a more difficult time. When you encounter difficulties again, there is always a kind of dispelling emotion, "the most difficult time has come, what are these?" "

On the other hand, the payment industry has benefited even more from the competition between the two top companies, and China's payment technology has become a dust-free technology in the world. At the end of 2017, the Spanish newspaper Le Monde expressed its views on China's payment industry, concluding: "China's payment revolution can be called one of the biggest technological innovations in China's history." "

On the evening of October 19th, 2018, Ant Financial Services Group held the HighMa Music Festival in Z Space | Oriental IC

The values of technology

In fact, before the "4-second shock" of Singles Day in 2010, Alipay technicians realized that the use of IOE commercial equipment (IBM- server provider, Oracle- database software provider, EMC- storage device provider, which make up an enterprise database system from software to hardware) and open source software can no longer meet the technical support requirements of the exponential growth of double 11 transaction volume. Especially when no one can completely presuppose the situation of that night. Even if it can be supported, the cost will be astronomical. Alipay decided to go to IOE, independently develop a distributed database, switch to cloud computing, and the OceanBase project was launched immediately.

Junyi remembers that the first technical transformation project he did on Alipay was to split the database. At that time, it was not because of double 11, but simply because the trading volume of Alipay website rose very fast, the database could not bear it, and the business could not be increased without dismantling it. This is in 2008.

In 2010, Junyi dismantled the database again. This time, he split the transaction database from the two databases that he dismantled last time into 10 minicomputers. By this time, it was almost the foundation for going to IOE. But soon, 10 minicomputers were not enough.

After the end of Singles Day in 2011, the connection between the application server and the database has reached a bottleneck, and there is no way to increase the capacity. In other words, the centralized and powerful single point of IOE can not meet Ali, especially the explosive business growth application model of Taobao at that time, and it also limits the exertion of technology potential. In addition, because IOE is a special equipment, there are separate design requirements for rack, power and network, and the cost pressure is already very great.

From January 2010 to July 2011 to complete the commodity warehouse to IOE (read-write separation, minicomputer, Oracle and EMC), and then to other core systems such as transactions to IOE, in 2013, Alipay the last minicomputer offline, I and E in IOE have been replaced by China's independent research and development technology, Shangyun has completed phased progress, which is like building an engine, which means that the trading volume of double 11 will no longer be restricted by technology.

However, in the first stage, it is still a bit of a chance whether the Singles Day holiday can be successfully passed every year. Starting in 2014, Alipay began to develop and implement full-link stress testing technology, which is a bit like a wind tunnel when building an aircraft, building a laboratory that completely simulates all the real environments at the peak of the day, and carries out stress tests on the system. According to Gong Jie, vice captain of the 2018 Great Promotion guarantee, the simulation of the real user request by the full-link pressure test can achieve more than 90% consistency with the double 11 request. As a result, on Singles Day, the probability of a smooth ride is extremely high, and the team's anxiety caused by uncertainty is greatly reduced.

Full-link stress testing, as a "killer" to eliminate uncertainty, has become a routine means of the current testing system. With the upgrading of the system, the frequency of use is also decreasing. Li Zheng remembers that when the full-link stress testing technology was just developed and used, "I wish I could do the test every day", but in the preparation for this year's Singles' Day, it is enough to do stress tests once or twice a week.

Alipay's double 11 is already a huge systematic project, and it is no longer possible to rely entirely on the human brain to solve all online problems. Therefore, Li Zheng feels that "intelligence" is another key word. The control of system engineering should be supplemented by such technical means as intelligent full-link pressure measurement, so that the problem can be solved more accurately and efficiently.

On November 2, the Great Promotion support Corps organized the last simulated full-link pressure test. Everything was ready, except Dongfeng, just waiting until 24: 00 on the 10th.

Miao Renfeng (4 from left) and Ant Financial Services Group colleagues at the scene of Singles Day office | the interviewees provided pictures

For payment technology, stability trumps everything, and stability means everything. As in previous years, the 10th year double 11, the importance of stability is still in the first place.

In addition to stability, the Alipay technical team has more to pursue. In the double 11 technical support in 2018, manual intervention has become less and less, because the whole support system is becoming more and more intelligent. For example, in previous years, when preparing for double 11:00, how much computing resources should be allocated and how to achieve the optimal configuration require very experienced engineers to carry out rigorous calculations, repeated stress tests and continuous tuning. But now the machine can be calculated and tuned automatically. For example, Cheng Li said that the payment guarantee of double 11 will be more and more toward the goal of "autopilot". Where to drive, where to stop, how to avoid risks and ensure safety are all intelligent.

The new changes are also reflected in the application of biometric payment and blockchain technology.

In Ni's discussion of marching, Alipay's understanding of payment tends to pay for disintermediation. In the end, payment does not need any carrier, and the human body itself is the largest medium. Of course, disintermediation cannot be completely separated, but biometric technology is the stepping stone for users to participate in digital scenes in the IoT era. Any scene system must first determine a problem of so-called digital identity, and people themselves are the best carrier. There is no need for other media to make a second switch. As a result, biometric is a technology that can reshape the experience.

According to Ni Xingjun, the payment proportion of biometrics (including fingerprint input, facial scanning, etc.) in daily application scenarios has exceeded half, reflecting the overall population's acceptance of the new payment experience corresponding to biometric technology. this signal makes him feel that the time has come to expand biometric users in life scenarios other than mobile apps.

In the first half of this year, biometric technology is really moving towards large-scale commercialization. Ni's expectation of marching is to achieve scale first, and on the basis that the terminal equipment reaches a million-level scale, according to the running-in of the connection between user behavior and various business scenes, and then consider the follow-up business demands. In the future, the application of new technology is bound to redefine the entire business process, and new millions of business opportunities will be born here.

This year, Tmall used blockchain technology to provide traceability of origin for 150 million cross-border goods, including large-value commodities such as diamonds from the Belgian Diamond Exchange.

Behind the change is Ant Financial Services Group's evolution and openness of BASIC technology strategy. Blockchain (blockchain), Aritificial intelligence (artificial intelligence), Security (security), IoT (Internet of things) and Computing (computing) these five clues constitute a clearer imagination for the future.

In the past ten years, Ant Financial Services Group's entire company has been constantly changing from centralization to distribution.

Personnel capacity has become more balanced. Junyi remembers that in the early years, at the critical moment of Singles' Day and many key technological problems, there were always several technical colleagues who came forward and showed extraordinary insight and ability at the moment, and finally passed smoothly. But now, Ant Financial Services Group's entire technical structure is becoming more and more complex, and it is necessary to form an overall and everyone-oriented engineering operation.

IT architecture changed from IOE to distributed, and then evolved into "offline hybrid". Last year, 25% were self-owned servers, 55% were on the cloud, and 20% were offline resources. This year, this proportion will be updated to 60% on the cloud, 20% online and 20% offline. In the meantime, offline computer rooms with poor performance can also perform online processing. The core lies in the further rational allocation of resources.

The distributed trend is gradually becoming a general trend: there are more and more computer rooms, expanding from Hangzhou to all parts of the country; more and more application systems and databases; and the team has expanded from the Alipay technical team to various product lines. In the past, the operation of the group was reliable, but now it needs to be decomposed layer by layer and organized as a whole to work together.

"from centralization to distribution" is a major feature of the form and content of social relations formed in recent years in the development of the Internet. If it is regarded as a value, as an Internet financial company with more than 51% of engineers, it is being deeply influenced, driven and changed. A large number of people, things and things in the enterprise are clearly showing this trend orientation. This enterprise worth hundreds of billions of dollars is also becoming a classic model of business, team innovation and development driven by technological values.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.