Shulou(Shulou.com)06/01 Report--

Architect group meeting: each issue selects one of the hottest technical topics to share practical experience.

The third phase: micro-service. Micro-service architecture is quickly concerned by architects and technical decision makers in various fields because of its high flexibility, flexibility and great improvement in efficiency. Its basic idea is to split a large system into several small service components, and the communication between the components is completed by a lightweight protocol. In this issue, we will discuss the micro-service practice of Internet companies.

Guests: JD.com Zhang Geng, former VIPSHOP Shi Tingxin, Seven Bull Chen Aizhen

This article is the arrangement of this exchange, which is divided into two articles.

The first round: free communication

JD.com Zhang Geng: Hello, everyone. I am Zhang Geng of the platform middleware of JD.com Infrastructure Department. I am mainly responsible for JD.com 's service framework, configuration center and other projects. JD.com carried out the technological transformation from. Net to Java at the end of 11 years, and put forward the concept of interface SOA at that time. At that time, JD.com 's business developed very fast, there were more and more projects, and service was an inevitable trend. JD.com 's service framework has a total of two generations. The first generation began 12 years ago, when we used an open source implementation, using Dubbo as the RPC framework and Zookeeper as the registry, which was supposed to be a mainstream technology choice. We have made some ease of use and functional extensions on the basis of open source, such as supporting REST, kryo/thrift and other serialization, service monitoring, etc., this framework is relatively stable under the business scale and the number of nodes at that time. At the beginning of 14 years, we began to develop our own service framework. Why would you do that? One is that the previous service framework still has a lot of room for improvement, whether it is performance or some functions of service governance, because JD.com 's machines and computer rooms are constantly being built, and the complexity of the network topology directly requires advanced service governance functions; another is the increase of the order of magnitude of interface nodes, when our interface scale has slowly increased by thousands, and hundreds of thousands of interface instances are also available. In addition, it is more troublesome to get through with some internal systems of open source, so we have to upgrade. At that time, I also considered whether to make changes with open source or to make all my own choices. Later, I felt that one was that I had time, and the other was that I felt that I had the ability to make a customized service framework that conformed to JD.com 's own, so we began to develop our own framework for 14 years and did it for a whole year. At the beginning of 15 years, we launched a new version of the service framework.

Our new registry, which is stateless, is based on the database for final consistency. Why choose the database to replace the previous Zookeeper? Because Zookeeper is a tree structure, querying it from some dimensions requires traversing the whole tree. The advantage of using the database is that it can be analyzed and filtered from multi-dimensional queries. No matter the computer room or IP, you can query, analyze, filter and compare the results. Then, Zookeeper has a problem when crossing data centers. Half of the nodes are alive before they can be used. Therefore, if deployed across data centers, at least three data centers are needed to ensure the overall availability of the cluster. There is also Zookeeper, which is strongly consistent and more dependent on the network. If the network is not good, if the network is cut off across the computer room, it is actually not available and cannot be read, so it will bring some problems. In the past, to register the server with zookeeper is to write a temporary node, which automatically disappears when the server is dead. But when you shake the network, or when there is a network failure between the server and the Zookeeper, it will accidentally remove it, because Zookeeper thinks it is dead, but in fact it is not necessarily dead, so it needs some assistance to judge whether it is really dead or not. The first situation we encountered was the temporary node. Later, we changed it to a permanent node, and then regularly telnet its port. For example, Dubbo directly executed the ls command, and we also directly executed a command to see if it responded. That was the first generation. In the JSF framework, there is a Sentinel component, our Provider will regularly send the heartbeat, for the registry, it knows the last heartbeat time, and then regularly brushes it into the database to see the situation of the Sentinel. If there is no Sentinel, the database stores a last heartbeat time. At this time, you can use a scheduled task to find it. This is our earliest version. Later, we improved it, because sometimes when the network was cut off, the heartbeat time was not correct. Therefore, there is an independent program in each computer room, and it is up to it to judge whether it is available at the same time when there is no heartbeat. If the program in the same room is not connected, we will set it into an unavailable state with a double guarantee. In addition, in the past, the list of services in Zookeeper was a full list of services, such as 1000 plus one, and 1001, while in the new registry, we pushed the change, adding only one, which greatly saved the amount of data pushed.

RPC framework, our internal name is Jeff, JSF, the abbreviation of JD.com service framework. We developed it ourselves based on Netty. In order to be compatible with the previous generation version and the previous dubbo protocol, we also use no code * *. We support multiple protocols on the same port, including custom JSF protocol, http protocol, dubbo protocol and so on. At present, we only have C++ and Java clients, and then if it is in other languages such as Golang,python,PHP, we will ask him to send a request to one of our HTTP gateways, which is mainly forwarded. Converting the previous http request to the latter JSF request is also based on Netty. Serialization defaults to Message Pack, which is a cross-language protocol and also supports hessian,json,Java,Protobuf and so on. The registry and the RPC framework are directly long-connected and can communicate with each other.

VIPSHOP Shi Tingxin: Hello, everyone. I'm Shi Tingxin. I was originally in JD.com and VIPSHOP, but now I'm in a hurry to see them off at home. I am basically in the area of warehousing and logistics. In the past, Asia No.1 was divided into modules according to the entire operational details of the warehouse, and each module was deployed separately. Each warehouse of storage, according to the category is different, the whole operation process is different. But the core things, such as inventory and order management, are the same, except that some combinations are different. For example, small items need to be put on and off the shelves, but for large goods, such as large household appliances, there is basically no need to put them on the shelves. You don't need to place an order. You don't need an order until you complete the delivery. In fact, you don't bind an order until the end. So each process is different and the combination required is not the same. At that time, we thought of taking each module apart first, mainly using KVM to run the entire node, and the business part would use these nodes to connect the whole. The core things, such as inventory, orders, commodity information, are basically using this simple module.

VIPSHOP's warehousing is also done in this way of thinking. At that time, VIPSHOP used Dropwizard, in fact, we used a shell of it, and then we still used Spring, and we also made a module to integrate Spring into one. Later, Springboot did it and basically abandoned Dropwizard. Because the Dropwizard version upgrade changes a lot. But it has an advantage, that is, Bundle is better, service split can be divided according to its size can be combined, if not split, we put the Bundle together, if we need to split, then split out the Bundle, alone this application is very easy.

I was sent to the home last year, our service registration, service discovery, gateways all use this set of Spring Cloud. So far, it's still good, and basically no major problems have been found. The only problem is that if the whole data extraction is not done well in the early stage, the whole report may be more troublesome.

Chen Aizhen: Hello, everyone. I'm Chen Aizhen from Qiniu. The design concept of the micro-service architecture is very suitable for the business characteristics of Qiniu, and each service is only responsible for a single responsibility. For example, audio and video processing services, such as audio and video transcoding, audio and video stitching, video frame thumbnails, and VOD streaming transcoding, are all built in the form of micro-services, so that each service has an independent running environment and can be independently expanded according to its own business pressure. Dynamic elastic expansion can also improve the utilization of resources. A landing micro-service architecture should provide an independent operating environment, scheduling framework, registry, configuration management center and monitoring platform for micro-services. Qiniu adopts the architecture of Mesos+Docker+ self-developed scheduling system. Docker does environment sealing, Mesos does resource scheduling, and the self-developed scheduling system is responsible for flexible scheduling of Docker.

We use Consul as a registry to realize the registration and discovery of services. Consul comes with key/value storage, which can be used for service discovery through DNS interface and specific health check function, and supports service discovery across data centers. API Gateway can query the list information of all available instances of the service through the DNS API provided by Consul, and forward the request. Traffic forwarding has the function of load balancing, using the way of polling, and service discovery is based on Consul. After the request comes in, the user queries the access addresses of all available nodes through Consul, and then sends the request to the back-end service for processing by polling. The returned results are only forwarded and interpreted and used by the requester. And there are monitoring components in the API gateway to monitor the number of requests, failures and so on, and transmit them to the Prometheus server. Through the monitoring data for flow control and service degradation and other corresponding processing.

When multiple micro services need to be called, we use pipeline for serial processing according to the business characteristics of Qiniuyun's data processing. The output of each microservice is the input of the next microservice, and the final data processing is not until the end of the execution of the last microservice. For example, after uploading a video resource, you need to perform two data processing operations: converting to mp4 resource and HLS slicing. In this scenario, two data processing operations cannot be performed on the original resource at the same time, but must be performed sequentially.

The second round: topic exchange

Host: how to manage the dependency between service and service? How to do the service arrangement? String these service nodes together.

VIPSHOP Shi Tingxin: for example, not business, basic services, such as picture servers, are independent. The only thing that is relevant is that there is a call between the business layer and the underlying basic service, and that there is a service call between the business module, which is not reflected in the system, but we add something similar to the default thing like a circuit breaker between the service and the service. We have made a simple system to maintain the dependency between systems. Spring cloud eureka is all stored in memory, we have improved, I have changed to a database. Our interactions are all Rest, and we wrote a set later, Google protobuf. The internal business modules of our warehouse are done with Google protobuf. We made a set of virtual ones on our own.

VIPSHOP Shi Tingxin: what I use now is Apache camel to do choreography, can use DSL description to string up the service. In fact, it is more about how the business process is organized and how to string the basic services together to form a really big business scenario.

JD.com and Zhang Geng: at present, most of JD.com 's one interface is already operating multiple tables, which is generally considered to be a relatively atomic operation. If there is another thing outside to wrap it, for example, wrapping three methods into one method, it doesn't make much sense, so we haven't done any work in this area now. In addition, JD.com has an elastic cloud and uses Docker, but it is not the Docker use of the kind of micro clothes you think. JD.com now uses something called "fat container", which is more like a virtual machine. What runs inside Docker is an application, and then this application, it may contain multiple interfaces and multiple methods. Maybe you understand that the Docker usage of microservices should be in a container, which only runs an independent logic, an operation on data, or an operation on a resource. For example, placing an order and withholding inventory, he may have run two containers in these two steps. Arrange it into a service of this, we don't seem to have this.

Host: how is the service split done?

JD.com Zhang Geng: JD.com 's current service is actually not too detailed, mainly because the business department controls the service granularity by itself. JD.com 's business is still relatively complex. Various applications depend on each other and call each other. Each operation basically involves a lot of resource changes, such as the operation on a single resource advocated by micro-services, basically none. To tell you the truth, the services in such large Internet companies are simply impossible, like RESTful, it is almost impossible to operate a put post on a resource. We are still determined by the R & D personnel who use our framework, and then generally they combine or split according to some relevance or atomicity or expansibility of this method. So we generally call ourselves service-oriented framework, not micro-service framework. In addition, they may also consider whether some services can be deployed independently, and they may consider these when dismantling them.

Host: in fact, many large e-commerce Internet can not be separated, can not be called micro-service, is that kind of service strength is very rough, and then each service number has several interfaces, in fact, the coupling is very high together, the correlation is very high, the data are all together.

JD.com and Zhang Geng: they are all the same. It is the business developers themselves who discount the service granularity. Like I just said, place an order. There must be a lot of tables to operate at a time. The business developer thinks that this is a few table operations, which is actually an atomic operation to place an order, so he breaks it down to such a fine point.

VIPSHOP Shi Tingxin: our service does not mean one department at a time. In fact, it is related to the data domain and is all put together.

VIPSHOP Shi Tingxin: according to the function, the call between the function and the function, if it is not in the same data domain, it is still called by RPC.

JD.com and Zhang Geng: that sounds wonderful. If you think we are dismantling it like this, there are already tens of thousands of interfaces, and the methods are basically over 100, 000. If you take it apart like this, you will at least multiply the interface by 10, and if you do so, the whole company will have to use a very great ability to do a good job.

Host: if he dismantles the kind of micro-service that is advocated in the market, in fact, it consumes a lot of internal services, because many of our internal calls to services are actually very frequent. If they are all taken apart to deploy independently, it actually requires very high consumption of the network.

VIPSHOP Shi Tingxin: it is the most difficult point. You can't take apart this data at first.

Host: as long as the data can be taken apart, you can do anything. As long as you can make the data granularity very fine, then there is no problem, that can also be done very fine, but it just can not be taken apart. In fact, a lot of data are old watches, are inherited from the original system, it is very difficult to disassemble.

VIPSHOP Shi Tingxin: so it's okay to make a new system. It's hard to make this old system plan. It takes many days to disassemble a meter.

JD.com Zhang Geng: and the amount of data is relatively small, and then the logic is relatively simple, such as the kind of entrepreneurial companies, the use of open source solutions, this can be very convenient.

Compere: in double 11:00 each e-commerce will definitely have large-scale promotion, at this time the pressure of service will certainly have a lot of growth, how to solve the problem of service scalability?

VIPSHOP Shi Tingxin: just like what we say on our side, it depends on which basis point has more pressure, and then you can spread a few more basis points. Now it's basically the Sino-Thai one, which is the Spring cloud thing. Expansion is always done in advance. Automatic expansion is equivalent to renting a simple monitoring system. According to his example, and because of the micro service, we are now Spring cloud, basically java-jar, and then we can start it ourselves. Anyway, you can put that jar in a public place. If you want to fight with remote java-jar, you can start it at any time according to his quantity during monitoring.

JD.com and Zhang Geng: JD.com 's service releases are all hung under the application, and the application release platform will actually communicate with various systems, such as database authorized systems, and then automatically hang up some VIP systems, monitoring platforms, log systems, and so on. Then we also have a monitoring platform to monitor some indicators, such as machine conditions, applications, as well as the performance of our own business sites. As for whether there is pressure on the service, it is their own assessment and expansion in advance. Generally speaking, a large-scale pressure test will be carried out before the big promotion, and they will do it by themselves, and then expand themselves according to the results. JD.com does not open automatic elastic expansion, basically is to greatly promote ahead of time to apply for the container, and then expand ahead of time.

VIPSHOP Shi Tingxin: businesses with large concurrency are placed in the public cloud. There are twice as many machines on Singles Day as usual. We only bought a cloud host for one month. Use it temporarily for a few days, and then it won't be necessary.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.