Shulou(Shulou.com)06/01 Report--

The architect group exchange meeting is a technical exchange meeting attended by technical experts from well-known domestic companies, and each issue selects one of the hottest technical topics to share practical experience.

This session invited Hujiang Java architect Huang Kai, VIPSHOP architect Zheng Minghua, Didi architect Zhao Wei, and Qiniuyun Technical Director Xiao Qin to exchange views on micro-service granularity, high availability and continuous interaction.

The first round: free communication

Huang Kai of Hujiang: Hello, everyone. I am Huang Kai, an Java architect from Hujiang. The first time I came into contact with the concept of microservices was in a Devops training at IBM three years ago, and I was impressed by the efficiency and inherent convenience of its development. Since then, I will consider the use of the concept of micro-services in my design.

After leaving IBM to join Hujiang, I just happened to reconstruct the Hujiang courseware project. In the demand analysis stage, it is found that the business is just in line with the characteristics of micro-services, using simple functions to connect complex businesses, combined with the service arrangement of Docker+Mesos+Marathon three Musketeers, so that we can greatly save the cost of operation and server, and from then on we love micro-services more. The architecture idea of Hujiang courseware system is very simple. If the requirements are separated according to the definition of resources, the CRUD of each resource will naturally become a micro-service. For example, the courseware information is regarded as a resource, which involves database resources and authentication service resources. If these three resources are made into micro-services and deployed in the production line environment, a natural resource invocation chain will come out. The granularity of each microservice is divided according to the teachings of the microservice guru Newman, so that it can be reconstructed in two weeks.

VIPSHOP Zheng Minghua: I am Zheng Minghua from VIPSHOP. We have not really implemented the definition of "microservice" in the strict sense, but we still want to divide all business areas vertically. Since I just arrived at VIPSHOP, this discussion is basically based on the experience of the last company. In the past, in Ali, we would split orders, commodities, members, payment, logistics, and later foreign trade-related customs clearance, tax rebate, financial settlement, and so on, into different services according to business areas. But we did not split the large services further, such as the reading service from the customs declaration system, or a certain part of the write split, did not do such a fine split. Just split a relatively large or relatively vertical domain, and then each domain is made up of one or more services.

At the beginning, we made the order issuing system into a large system, but then we found that all the business logic of this order issuing system was in it, O2O in it, value for money in it, Taobao in it, and Tmall in it. Later, we took out the cost-effective ones, took O2O out, and split them into micro-services one by one. Then take out the various transaction rate readings of Taobao and Tmall and tear them down one by one. Therefore, the evolution of services is changed step by step with our business and the understanding of the system, which is not to say that a design is a big one, but it is actually broken down into a lot of small ones.

I used to work in Taobao and Tmall. We are all split into a service, just like Didi's. In the past, the buy system was also a big system, and our buy system was a big service system. Later, the entire buy system is made into a similar business, different business is a different service. There is also a TP system, which does a public service to accept all orders. This is what we call the trading platform system.

Didi Zhao Wei: Hello, everyone. My name is Zhao Wei. I come from Didi. At present, I am in charge of the structure of the Didi Division. Micro-service should be an architectural model, which divides a system or an application into small services, which communicate with each other, cooperate with each other, and ultimately provide value for users.

Microservices have several characteristics:

Each service is a single process

The business is independent.

Each service only does a single function.

Each service is highly autonomous, and from its development, testing, including construction and deployment, it is independent and independent of its own set.

We have adopted the micro-service architecture because of the following advantages:

We think that micro-service is relatively simple. If a business wants to consider it in a unified way, it is a complex problem, and it can be divided into complex problems through micro-service. It is a process of downgrading the difficulty.

The good thing about it is that the service is an abstract thing, it is a kind of reuse.

Its advantage is that it is transparent. For the caller, it only needs to care about the calling interface. In fact, I do not care about the complexity of the business logic implemented within the service.

Scalability will be better, because we now through the service stateless design to facilitate the unlimited expansion of services to support large-scale Internet requests.

Then another advantage is the transformation, which is reflected in two points. The first point is that it is more convenient to try some new technologies. As it has been dismantled into a lot of services, if there is a new technology you want to try, you can actually find a three-tier system or a second-tier system, you can try this technology. If the service can be achieved successfully, it can be promoted on a large scale in the whole system. In this process, the impact on the business will be minimized, and this uncontrollability will be minimized. On the other hand, architectural transformation will be more convenient, such as service split or service merge, which will be relatively simple.

The last advantage is that it is easier to release than a single service, because each service has already been split, and its dependence on the outside, or the environment, will have less impact on it after the split, so it is relatively simple to release. And the driving division is now a stateless service, and all the states are stored in the cache. Therefore, based on these advantages, we are designed according to the micro-service architecture at Didi Didi.

Qiniu Xiao Qin: Hello, everyone. I am Xiao Qin, Technical Director of Qiniu. I am currently in charge of infrastructure operation, deployment and R & D, and provide support to colleagues in the infrastructure department. The design concept of the micro-service architecture is very suitable for the business characteristics of Qiniu. Each service is responsible for a single responsibility. For example, audio and video processing services, such as audio / video transcoding (avthumb), audio / video stitching (avconcat), video frame thumbnail (vframe), and VOD streaming transcoding (avvod), are all built in the form of micro-services, so that each service has an independent running environment and can be independently expanded according to its own business pressure. Dynamic elastic expansion can also improve resource utilization.

When multiple micro services need to be called, we use pipeline for serial processing according to the business characteristics of Qiniuyun's data processing. The output of each microservice is the input of the next microservice, and the final data processing is not until the end of the execution of the last microservice. For example, after uploading a video resource, you need to perform two data processing operations: converting to Mp4 resource and HLS slicing. In this scenario, two data processing operations cannot be performed on the original resource at the same time, but must be performed sequentially. At the same time, it also supports the definition of service groups for some commonly used MServer collections, such as transcoding and watermarking all videos, so that the two services can be defined as a service group, so that two services can be executed at the same time each time the service group is called.

The second round: topic exchange

Host: what is the granularity of microservices? What is the corresponding relationship between each microservice and the database and service data?

Didi Zhao Wei: with regard to the micro-service granularity, my understanding is that what is suitable for me is the best. In fact, there is no stipulation about the granularity of the service, how much code it takes or how long it will take to develop. In fact, there is no such thing. When the system is just launched, according to the business cooperation, in addition to the business, may also be based on the organizational structure of this way to cooperate with micro-services.

At first, when we went online, we had only one business scene, that is, drunk driving. At that time, our Team was divided into five Team: management background, order, user, marketing and payment. Then each Team, the user's word is the user service, and the order word is the order service. In addition to the main services, there are also some secondary services, such as composite services. At that time, the strength of our service was relatively rough, such as the order service from the user's order, finally to dispatch order, rush order, the whole set of service is all in it.

With the development of our business, our business scenario is constantly expanding. In addition to the latest maintenance program, the biggest difference between maintenance and drunk driving is that drunken driving users actually need a driving service when they place an order. But on behalf of maintenance, the next order actually requires two driving services. So in the original order of the service, you will find that the granularity of its service is no longer appropriate. The reason is that in this business scenario, it is very difficult for you to add to the original order, and the maintenance cost is very high, and then we split the granularity of the service. For example, we split the fine-grained service of dispatching or grabbing orders into a pool of basic services. In the basic service pool, there are orders, billing, full drivers, dispatches, including real-time billing, all of which were originally in the same order service. But now that it is broken down into basic services, we will all have special personnel to maintain it, and if we want to change it, we will have special personnel to change it.

We call orders, payments, users and so on general services. Ordinary services correspond to business scenarios, such as drunken driving, which has its own orders and uses basic services, such as return orders and full drivers. Generation maintenance will also have its own orders, because this kind of order may not be the same as that of drunken driving, but its business form of issuing orders, full drivers, and dispatching orders is the same, so it will also use the basic services in our basic service pool. Because the service granularity is reduced again, it is equivalent to some basic services abstracted from different business scenarios, and then provide different business scenarios. The evolution of the architecture is gradually evolved with the development of the business form, and it is impossible to be in place at once. In every business, it has a business time, and if you miss the window, you may miss it. So you have to put the system up first and evolve step by step in terms of architecture. At the beginning, we only had drunken driving, and you had to consider what kind of maintenance, what bag driver and so on, which was unnecessary at that time, and you didn't know what the final demand would look like.

Hujiang Huang Kai: in fact, there are some criteria for micro-service. For example, it is best to complete it within two weeks. But in the actual project, we have to consider three points. The first one is Share requestful. What does that mean? Concentrate your functions on yourself, and separate as much as possible those that don't belong to these functions. This is what we often call high cohesion and low coupling. The second is whether it can be independent and autonomous. The last one is whether it can be released automatically. I think only by meeting these three points can we call it a micro-service.

VIPSHOP Zheng Minghua: this grain size is a bit fine. It is a good principle to do academic research, but in terms of engineering, I think it still needs to be reconsidered. It is still too fine.

Hujiang Huang Kai: agree, in fact, because these granularities are too fine, leading to a problem. Even what you have just discussed, I think it is reasonable, reflecting a Conway law. Conway's law says that an organizational structure determines the system structure. If the organization of the company is not divided into such details, no matter how fine the architecture is designed, it is impossible to achieve, because I think the system architecture can not determine the organizational structure. But if our organizational structure slowly becomes smaller, then the system structure will certainly develop to micro-services.

VIPSHOP Zheng Minghua: another is that I think that if the service is broken down too carefully, it will be difficult to solve the problem of service management and distributed service.

Huang Kai of Hujiang: as far as I know, Tencent has a micro-service structure. The call chain reaches 36 levels. It takes 36 layers to call a request before giving a response. I think this is too detailed. That's what WeChat red packet did. They pay at a rate of 156000 per second. They spent too much money on calling between gateways and services. Tencent basically optimized the underlying layer, even using the cache of the network card. Tencent has achieved the extreme of system performance mining and tuning, 36 layers of call chain, that is, for a while, did not see WeChat red packet collapse during the Spring Festival.

VIPSHOP Zheng Minghua: Tencent has a buddy who is a Linux kernel developer, so he should be able to change the cache of the network card you mentioned, but it is very difficult for ordinary companies to have experts in this field to do this.

Didi Zhao Wei: Tencent's operating system has actually been optimized by itself, because if they are in the mainstream of Tencent, they are very familiar with Linux,TCP. And their bottom layer is unitary. Although they have more than 30 layers of call chains, they are all done in the same computer room or in the same area, and they request that they will not generate cross-room or cross-regional calls. Moreover, their own internal machines are also very good, and the bandwidth is also very adequate. And it lies in the network transmission between services, which is not time-consuming, and may mainly be in the business, that is, the market problems that each service deals with on its own.

Hujiang Huang Kai: I think this should be based on their own business and their own R & D capabilities. many people's understanding of micro-service is a single service, a slimming version of a fatter single service, losing some and excluding some functions.

VIPSHOP Zheng Minghua: it's not that simple. First of all, you should split the business from the beginning of the business to the underlying database.

Hujiang Huang Kai: but this split is very painful. It's fine if you start to design micro-services. If you start as a large-scale service, especially a financial business like banks, it's even more difficult to split into micro-services.

VIPSHOP Zheng Minghua: it seems that banks do not use this model of micro-service. The bank's database is still a single database, or an Oracle database, or a mainframe.

Hujiang Huang Kai: I remember an architect in the development bank once told me that they didn't want to move, they didn't dare to move, because no one knew what was written in his module. If they make a slight change, and then the transaction or the transfer goes wrong, who is responsible?

Didi Zhao Wei: I have a few questions to ask.

The first question is, when you dismantle the service, will you make some adjustments to your organizational structure, because you have broken it down into micro services? how can this kind of team be responsible and responsible to people? I don't know what you think.

The second question is that in the whole microservice release process, it is a process that can not be done at once. How can you ensure that it will not affect the development of this kind of business? Otherwise, if you are adjusting the structure, how long you say you don't pick up the demand or whatever, the business side will jump up.

The third problem is that unlike a single service, a single service loses everything. In fact, we can perceive it very quickly when something goes wrong. Once there is a problem with the microservice, it will feel very small at first, but the problem will be magnified like the snowball effect, resulting in the unavailability of the service. How do you think about monitoring, alarm, demotion and so on?

VIPSHOP Zheng Minghua: the first question you said is very sensitive. The system architecture will directly affect the organizational structure. But I very much do not want to see such a problem that the organizational structure affects the system architecture. I hope that the system architecture affects the organizational architecture, so the problem that I faced in the previous team is more difficult to solve. Very often, I hope that the boss will support me and adjust the balance of interests between departments. Sometimes, the architecture design needs to make some compromises.

Didi Zhao Wei: yes, but because he is your supporter, it would be very difficult and there would be a lot of resistance without this good support for the whole landing of your micro-service.

Qiniu Xiao Qin: the micro-service architecture in Qiniu is now an imperceptible influence. The microservice architecture describes not only the technical architecture, but also the team architecture. Just like a spirit of service, you should pay attention to the construction, operation and management of this service, which is very beneficial in the team. Everyone can have a clearer understanding of their responsibilities, thus giving full play to their subjective initiative. Including operation, later improvement, can be improved spontaneously, team management will be easier and more efficient.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.