Shulou(Shulou.com)06/02 Report--

This article mainly explains a solution to the problem of the explosion of Redis connections in Docker, the content is clear, interested friends can learn, I believe it will be helpful after reading it.

The redis server is unavailable on the production server on Saturday. The error message is:

The status is not available and cannot be used until the background checker resumes. Unexpected end of stream; expected type 'Status'

As shown in the following figure, figure 6300 below is the port on which our redis server is running.

The first time I encountered this kind of problem, I wondered if redis hung up and then passed through the telnet ip+ port. Find that it is working properly, and then want to go to redis to see the current connection. I found that there were as many as 1903.

Then look at the code, thinking that it should be caused by too many redis connections created by the code.

It is found that this is the only place where the redis is created, and it is also executed when the service is registered. That is, it is executed only once when the application starts. Then look for the entire project, and there is no other place to call redis initialization.

Is it possible to create a connection every time you read and write data in redis? Does it have anything to do with frequent reading and writing? It always feels like it won't, and then create test code to test it.

Set up a redis environment locally. Check the number of connections before testing. At present, there is only one, that is, the current cmd connection client, which is normal.

Start the test and run the program. The code is to create a connection object and test a total of 1000 writes and 1000 reads.

No matter how much I test, there are 6 connections, which means that our program has created at most 5 connections, mainly with thread pools in it.

So it must be no problem for basic storage to read this code.

But this part of the code doesn't completely give up troubleshooting, because the production server runs about six applications through docker. Are all connected to the same redis, could it be caused by other applications?

Then you want to directly query the corresponding process information through any port in the redis connection list to know which applications it is.

Process information is displayed in Linux by querying the network port number.

Netstat-atunlp | grep 60852

First of all, take a look at the IP corresponding to this port, for example, the first one here is 172.17.0.1. Students who are familiar with docker should know that this ip is the docker gateway IP. All the programs in our container communicate with our host host through this gateway IP. We can find the gateway IP of docker through ifconfig, and the second 172.17.0.3 IP 6379 is the container IP of redis at first glance.

From this point of view, it is really impossible to find the specific container in which the program is connected to us.

One of the stupidest ways is to enter the container one by one. Docker exec-it test / bin/bash and then check the network connection of the current container. This is very troublesome and requires a lot of components to be installed to execute a series of commands.

Alternatively, the lsof command, if not, needs to be installed. We can find all the network connections through the process.

For example, we just found out that our process is mainly docker, and his pid is 582251.

Lsof-I | grep 582251 or lsof-I-p 582251

As the result is shown in the figure below, there is actually a specific IP on the right. This IP is the specific IP address of the docker container.

Now that we know all the IP and ports, we download the command execution results.

First find your own IP for each container.

Docker inspect name | grep IPAddress / / name container name or id

Find each ip and then make statistics based on all the network connection information you just downloaded to see which IP has the most connections and which one must have a problem.

Then I find the program deployed by the container corresponding to the IP and look at the redis configuration. Found that the thread pool is set to 200.

In addition, through github, I found that CSRedisCore also has a preheat mechanism, that is, preheat, its default value is 5 prefetch connections.

Our thread pool is set to 200, plus we have a warm-up mechanism for 5 connections. I don't know if we will create 200, 5, 000 connections. If you have time to study the source code, it's just a guess.

I have now changed redis to poolsize=5 preheat=false. 5 thread pools, and turn off the warm-up mechanism.

After modifying our connection configuration and restarting the application server and the redis server (to completely clear the established connections), we found that the number of connections decreased, but not much. Later, the query found that it was redis's idle that was idle for too long, causing the connection pool to maintain too many connections and not be released.

Let's set the timeout to 30s.

Execute CONFIG SET timeout 30 (in seconds), which is only a temporary modification and is valid for the current run. Long-term remember to modify redis configuration file)

Then take a look at the number of connections, which reduces a lot at once.

Summary:

1. Redis connection explosion, first of all, start from your own application to find the problem, for example, I found that the connection pool setting is too large, coupled with the default preheating mechanism. Also try to see if the code level will be triggered multiple times when creating a connection, and if so, it must be corrected. Now instances are created by injection, depending on whether the place is called multiple times.

2. Modify the redis server configuration, such as connection idle timeout. Including can also look at the maximum number of connections, the default value.

After reading the above, do you have a better understanding of the solution to the problem of a surge in Redis connections in Docker? if you want to learn more, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.