Shulou(Shulou.com)05/31 Report--

Many novices are not very clear about how to build micro-services based on Go technology stack. In order to help you solve this problem, the following editor will explain it in detail. People with this need can come and learn. I hope you can get something.

In the micro-service construction of large-scale systems, a system will be divided into many modules. These modules are responsible for different functions, combined into a system, and can eventually provide rich functions. In this form of construction, developers generally focus on the maximum decoupling of the functions of modules to reduce the additional development costs caused by inter-module coupling. At the same time, micro-services are faced with new problems such as how to deploy these large number of service systems and how to operate and maintain these systems.

The material of this paper comes from some best practice cases in our development, and introduces some of our micro-service construction experience based on Go technology stack from the point of view of development, monitoring, logging and so on.

Start to send

In the development process of microservices, different modules are taken care of by different developers, and clearly defined interfaces help to determine the tasks of developers. In the final system, a business request may involve multiple interface calls, so how to call the remote interface accurately and clearly is also a big challenge. For these problems, we use gRPC to be responsible for the protocol formulation and invocation.

Traditional micro-services are usually based on http protocol for inter-module invocation, but in our micro-service construction, we choose the gRPC framework introduced by Google to invoke.

The interface of gRPC needs to be defined by Protobuf3 and statically compiled before it can be successfully called. This feature reduces the communication costs due to interface changes. If you use http rpc, you need to change the interface document first, and then inform the caller. If the caller does not modify it in time, the error will probably not be found until the service is running. In this mode of gRPC, errors caused by interface changes are guaranteed to be eliminated at compile time.

In terms of performance, gRPC has a significant improvement over the traditional http rpc protocol (according to this assessment, gRPC is 10 times faster). GRPC uses http 2 protocol for transmission. Compared with http 1.1, http 2 multiplexes tcp connections, reducing the overhead of establishing tcp connections per request. It should be pointed out that if the pure pursuit of performance, the industry will generally choose the rpc protocol built on the tcp protocol (thrift, etc.), but the layer 4 protocol can not easily do some transmission control. In contrast, gRPC can put control fields in http header, together with nginx and other proxy servers, can easily achieve forwarding / grayscale and other functions.

Next, let's focus on how we use some of the features of gRPC in practice to simplify the development process.

1. Use context to control the lifecycle of a request

In the go language implementation of gRPC, the first parameter of each rpc request is context. The http2 protocol puts the context in the HEADER, and as the link passes, you can set an expiration time for each request, and once a timeout occurs, the initiator ends the wait and returns an error.

Ctx: = context.Background () / / blank context

Ctx, cancel = context.WithTimeout (ctx, 5*time.Second)

Defer cancel ()

Grpc.CallServiveX (ctx, arg1)

In the above code, the initiator sets a waiting time of about 5s. As long as the remote call does not return within 5s, the initiator will report an error.

In addition to adding timeouts, context can also add other content, and we'll see another wonderful use of context below.

two。 Using TLS to realize access Control

GRPC integrates the TLS certificate function, which provides us with a very perfect access control scheme. In practice, suppose that service An exists in our system, and because it is responsible for manipulating the user's sensitive content, we need to ensure that An is not abused by other services in the system. In order to avoid abuse, we design a set of self-signed secondary certificate system. Service A holds the self-signed root certificate and issues a secondary certificate for each service that invokes A. In this way, all services calling A must be authorized by A, and A can also identify the caller of each request, so it is convenient to do some logging, flow control and other operations.

3. Use trace to track requests online

GRPC has a built-in trace system to track requests, which can not only track the detailed log information of the last 10 requests, but also record the statistics of all requests.

When we add the trace log to the request, the trace system records the last 10 requests for us. The example shown in the following figure is the addition of business data tracking to the trace log.

Macroscopically, the trace system records the statistics of requests for us, such as the number of requests, the distribution of statistics according to different request times, and so on.

It is important to note that this system exposes a http service that we can turn on or off on demand at run time through the debug switch to reduce resource consumption.

Monitor and control

1. Determine the monitoring index

When we received the task of building a monitoring system for the entire system, the first question we faced was what to monitor. GoogleSRE's book provides a detailed answer to this question, and we can monitor four gold indicators, namely delay, traffic, error and saturation.

The delay measures the time spent on the request. It should be noted that considering the long tail effect, it is far from enough to use the average delay as a single indicator of delay. Accordingly, we need 90%, 95%, 99% of the median delay to help us understand the distribution of delay. A better way is to use histogram to calculate the distribution of delay.

Traffic measures the request pressure on the service. The traffic statistics for each API can let us know the hotspot path of the system and help optimize.

Error monitoring refers to the statistics of the results of incorrect requests. Similarly, each request has a different error code, and we need to make statistics for different error codes. Combined with the alarm system, this kind of monitoring allows us to perceive errors and intervene as soon as possible.

Saturation mainly refers to the load monitoring of system CPU and memory. This kind of monitoring can provide a basis for our decision to expand capacity.

two。 Monitoring and selection

When choosing the monitoring scheme, we are faced with two main choices, one is the company's own monitoring system, and the other is to use the open source Prometheus system to build. The differences between the two systems are listed in the following table.

Considering that our entire system has about 100 containers spread over 30 virtual machines, the stand-alone storage of Prometheus is not a bottleneck for us. We do not need to keep the historical data completely, and the greatest advantage of the self-built system is not enough to attract us to use. On the contrary, because we want to be able to count many indicators generated by the four major gold indicators, the convenient DSL of Prometheus can greatly simplify our index design.

Finally, we chose Prometheus to build the monitoring system. The framework of the whole monitoring system is shown in the following figure.

Each service registers its own address in consul, and Prometheus automatically pulls the target address to be monitored from consul, then pulls the monitoring data from these services and stores them in local storage. In the Web UI that comes with Prometheus, you can quickly use PromQL query statements to obtain statistical information. At the same time, you can also input query statements into grafana, fixed monitoring indicators for monitoring.

In addition, with the plug-in AlertManager, we can write alarm rules to send alarms to mobile phones / emails / mailboxes when there is an exception in the system.

Journal

1. Log format

One problem that is often overlooked is how to choose the format of logging. A good log format is conducive to the subsequent tools to cut the log content and facilitate the index of log storage. We use logrus to print logs to files, and the log format supported by the logrus tool is wrapped in space-delimited single-line text format, json format, and so on.

Text format

Time= "2015-03-26T01:27:38-04:00" level=debug g = "Started observing beach" animal=walrus number=8

Time= "2015-03-26T01:27:38-04:00" level=info msg= "A group of walrus emerges from the ocean" animal=walrus size=10

Json format

{"animal": "walrus", "level": "info", "msg": "A group of walrus emerges from theocean", "size": 10, "time": "2014-03-10 19 Vera 57 info 38.562264131-0400 EDT"}

{"level": "warning", "msg": "The group's number increased tremendously!", "number": 122,122,122, "omg": true, "time": "2014-03-10 1957 The group's number increased tremendously 38.562471297-0400 EDT"}

two。 Call log collection on end-to-end link

In the micro-service architecture, a business request will go through multiple services, and collecting logs on the end-to-end link can help us to determine the specific location of the error. In this system, we generate a global ID at the request entrance and pass the ID over the link through the context in the gRPC. The logs of different services are collected into graylog, and the logs on the entire link can be queried through an ID.

What is gogo is the abbreviation of golang? golang is a static strongly typed, compiled, concurrent and garbage-collected programming language developed by Google. Its syntax is similar to C language, but it does not include functions such as enumeration, exception handling, inheritance, generics, assertions, virtual functions and so on.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.