Shulou(Shulou.com)06/03 Report--

This article mainly introduces "how to solve the problems of writing interface". In daily operation, I believe many people have doubts about how to solve the problems of writing interface. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubt of "how to solve the problems of writing interface"! Next, please follow the editor to study!

Late at night, the leader: "there is something wrong with the interface you wrote! get up and have a look." Ding! As soon as the reminder software rings, you will know that it is time for Work.

But after thinking about it, I think it's impossible. My code, is a simple Redis query ah, can it be Redis failed?

Colleagues sent all the evidence to the group, it is your interface no doubt. A simple Get query takes an average of 2 seconds. Jstack,promethus monitoring points all the problems to your interface!

Log in to the Redis server and everything is fine. What should I do? Do you want to recite a big iron pot of Zhangqiu so inexplicably?

Quick is the original sin.

In this case, trust your instincts. Your interface is fast and good, it is likely that the wood is better than the forest, stand out from the crowd, and become the scapegoat.

In "some" and "high concurrency" environments, because resources are not isolated, the performance of some logs and tools can be very confusing when problems occur.

The interface that causes problems is the fastest and most requested interface, but it is theoretically impossible.

As pictured above. This situation is very common. Most requests, through the scheduling of the Tomcat thread pool, do real business processing.

Of course, the thread pool does not do this dirty work, and it hands the request to the resource processing pool, such as:

A database connection pool that performs time-consuming statistical operations and rapid query operations.

A Redis connection pool that performs blocking slow queries and simple GET SET.

A Http connection pool (HTTPClient, OkHTTP, etc.) that remotely invokes resources with varying speeds.

...

In our usual coding, we usually share such a resource pool. Because it's easy to write code and doesn't need to use your head.

But if your service itself is not split and isolated, the problem can be fatal. For example, you put the report interface and the highly concurrent C-side interface on an instance.

At this time, you may be cheated by the report interface.

An example

Let's take database connection pooling as an example to illustrate this process. Let's take a look at the following basic information:

Connection pool for Tomcat, with a configuration size of 200.

The connection pool of MySQL is relatively large with a configuration size of 50.

Interface A needs to call a time-consuming query, which takes 5 seconds.

Interface B is very fast, and the response time for querying the database is below 200ms.

For the fast B interface, the number of requests is much larger than that of interface A, and it is usually at peace.

One day, interface A suddenly had a large number of queries. Because it took a long time, it quickly filled up 50 connection pools in the database (interface B will be eaten by A slowly because of its fast response and short holding time).

At this time, both interface An and interface B requests need to wait at least 5 seconds to obtain the next database connection before the business can go on normally.

After a while, the state of the service looks like this:

There are 50 connections in the database connection pool, which are filled quickly and almost all of them are filled with slow queries.

The 200 connections in the Tomcat connection pool are quickly filled up, most of which are fast interface B because of its large number of requests and high speed.

All interfaces are Block on the thread of Tomcat. As a result, even a request to query a non-database will have to wait about 5 seconds.

In general, when we encounter this kind of problem, we tend to use jstack to print the information stack or look at some internal monitoring curves.

Unfortunately, most of this information is deceptive, and the slow query you see is not really slow query.

From the above analysis, you should easily see the crux of the problem: unisolated bottleneck resources cause a chain reaction of upstream resources.

But in my normal work, I have seen a classmate scramble about it more than once. A lot of evidence points to fast and good interfaces that have nothing to do with them at all. Their happy screenshots, @ related people, etc., are extremely arrogant.

In this case, you can use the following script for preliminary analysis:

$cat 10271.tdump | grep "waiting to lock" | awk'{print $5}'| sort | uniq-c | sort-K1-r 26 18 16 10

In the above example, we find the execution stack that locks 0x0000000782e1b590, and we can find that it is all stuck on the read operation of HttpClient.

In the actual scenario, you can look at several lock addresses at the top of the list to find out the commonalities:

These stacks, which show very little information, are the root cause of the problem.

How to solve

Increasing the size of the Tomcat connection pool, or increasing the size of the connection pool, will not solve the problem, and the high probability will happen again.

The best solution, of course, is to separate time-consuming services from normal services, such as popular microservices. Your service query is slow, your own access timed out, and my service has nothing to do with it.

However, the fact that your service can encounter this kind of problem proves that your company lacks the conditions for such transformation. We can only make a fuss about the single service.

This kind of practice is isolation:

As shown in the figure above, we have created two MySQL database connection pools in the same project, pointing to the same MySQL address.

In this way, the operation of connection pooling can be relatively independent of each other. But so far, it's not over, because your Tomcat connection pool is still shared.

Slow query related, the strategy of obtaining connections from the connection pool should be changed. You can't wait all the time. Instead, you should use FailFast (it is also possible to get a short timeout of connections), otherwise the symptoms will remain the same.

The current popular concept of circuit breaker also practices this isolation to a certain extent.

At this point, the study on "how to solve the problem of writing interface" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.