Shulou(Shulou.com)06/03 Report--

Preface

In the actual project, I have encountered the peak of online 5W+QPS, and also experienced the heavy traffic request of 10W+QPS in the state of stress test. The topic of this blog is mainly my own thinking about high concurrency traffic control.

Some thoughts on how to deal with large flow

First of all, let's talk about what is large traffic?

With a lot of traffic, we are likely to pop up: TPS (transactions per second), QPS (requests per second), 1W requests per second, 5W transactions, 10W transactions, 100W transactions. In fact, there is not an absolute number, if this amount causes pressure on the system and affects the performance of the system, then this amount can be called large flow.

Secondly, what are some common ways to deal with large traffic?

Caching: to put it bluntly, let the data get into the cache as soon as possible, be close to the program, and don't access DB frequently.

Downgrade: if it is not the core link, downgrade the service. To make an analogy, today's APP pays attention to thousands of people. After getting the data, do personalized sorting display, if in large traffic, this ranking can be downgraded!

Current restriction: as we all know, Beijing subway morning rush hour, subway stations will do one thing, that is, current restriction! The idea is very straightforward, that is, to limit the request to a certain range within a certain period of time, to ensure that the system will not be washed out, and to improve the throughput of the system as much as possible.

Note that sometimes caching and downgrading cannot solve the problem. For example, e-commerce's Singles' Day, users' purchases, orders and other behaviors involve a large number of write operations, and the core links cannot be degraded. At this time, current restriction is more important.

Well, next, let's focus on current restrictions.

Common ways of current restriction

The common treatment methods of current limitation are: counter, sliding window, leaky bucket, token.

Counter

Counter is a relatively simple current-limiting algorithm, which has a wide range of uses. at the interface level, this method is used in many places. In a period of time, count, compare with the threshold, to the time critical point, the counter will be cleared 0.

It should be noted here that there is a time tipping point. Take Chestnut, there are no user requests between 12:01:00 and 12:01:58, then 100 requests are made at 12:01:59, OK, and then 100 more requests are made at 12:02:00. Here you should be able to feel that there may be a large number of requests from malicious users at this tipping point, even more than the system expects.

Sliding window

Because of the critical point defect of the counter, the sliding window algorithm appeared later to solve the problem.

Sliding window means that the fixed time slice is divided and moved as time goes by, thus cleverly avoiding the critical point problem of the counter. In other words, these fixed number of movable grids will count and determine the threshold, so the number of grids affects the accuracy of the sliding window algorithm.

Leaky bucket

Although sliding window effectively avoids the problem of time critical point, there is still the concept of time slice, and the leaky bucket algorithm is more advanced than sliding window in this respect.

There is a fixed bucket, the rate of water intake is uncertain, but the rate of effluent is constant, and it will overflow when the water is full.

Token bucket

Note that the outlet speed of the leaky bucket is constant, which means that if there is an instantaneous high flow, most of the requests will be discarded (that is, the so-called overflow). In order to solve this problem, the token bucket algorithm is improved.

The speed at which tokens are generated is constant, and there is no speed limit for requests to get tokens. This means that in the face of instantaneous heavy traffic, the algorithm can request a large number of tokens in a short time, and the process of getting tokens is not very expensive. (a little bit of production token, consumption token)

Whether the token bucket can not get the token is rejected, or the water of the leaky bucket is full of overflow, it is reasonable to sacrifice a small part of the traffic in order to ensure the normal use of most of the traffic. if a very small part of the traffic needs to be guaranteed, then it may cause the system to reach the limit and die, and the loss outweighs the gain.

Current limiting artifact: Guava RateLimiter

Guava is not only powerful in collection, caching, asynchronous callback, etc., but also encapsulates current-limiting API for us!

Guava RateLimiter is based on the token bucket algorithm, so we only need to tell RateLimiter how many QPS is limited by the system, then RateLimiter will put tokens into the bucket at this speed, and then, when requested, use the tryAcquire () method to obtain a license (token) from RateLimiter.

Current limitation in distributed scenarios

Some of the current-limiting methods mentioned above are for a single machine, in fact, in most scenarios, the current-limiting of a single machine is enough. The means of distributed lower current limit often require the combination of a variety of technologies, such as Nginx+Lua,Redis+Lua and so on. This paper mainly discusses the current limit of a single machine, so we will not introduce the current limit in distributed scenarios in detail here.

In a word, let the traffic of the system be queued and limited in the queue first, and do not let the traffic call directly to the system.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.