Shulou(Shulou.com)06/01 Report--

When viewing redis monitoring, you can see that the graph of redis is abnormal. The screenshot is as follows:

As shown in the screenshot above, you can see that the number of client connections to redis rises abruptly to 10K and suddenly drops to 0. 5%. The reason for the monitoring itself is ruled out, obviously because there is something wrong with redis itself.

Go to the redis server and connect to it

/ usr/bin/redis-cli-p 6399-h 127.0.0.1

> 127.0.0.1 6399 > > info

ERR max number of clients reached

Regardless of the execution of the command, the error above is displayed. The only thing I can think of at this time is that the client of redis has reached the maximum number of connections and cannot create a connection. That is, the file descriptor that redis client can open is insufficient.

1. Get the PID information of the service redis:6399

Netstat-tunlp | grep 6399

two。 Gets the maximum file descriptor that can be opened by redis

Cat / proc/PID/limints

You can see the maximum file descriptor that the process can open by looking at the parameter Max open file in the screenshot above.

3. View file descriptors opened by the process

Method 1: ll / proc/6677/fd | wc-l

Method 2: lsof | grep 6399 | wc-l

The values obtained by the above method are 10015 and 10007, and you can see that the file descriptor for redis has been used up.

4. Because the above error occurs after the redis is connected to the redis-cli, the above error occurs, but because the redis server is used as a cache, it cannot be restarted casually, so it can only find a way on the client and can only actively disconnect the client. So get the connection information for port 6399:

Netstat-tun | grep 6399 | awk'{print $5}'| awk-print $5}'{print $1}'| sort | uniq-c

8 10.143.106.95

8 10.143.90.216

5433 10.207.169.67

4788 10.221.236.76

1 10.221.244.39

6 10.225.36.189

The command above can obviously get a lot of connections to that IP address to the redis service.

5. Go to the server with the most connections and get the client information that connects to redis

Netstat-tun | grep 6399

Lsof-i:Port

According to the above two commands, you can get the application service information of the client.

6. After obtaining the application service information, we discussed with the developer, we can only restart to allow the client to actively disconnect. So the application service was restarted directly.

7. Change redis configuration information.

After restarting the client, the connect of redis has also been reduced, and the redis can be connected at this time.

A) obtain the status information of redis

> 127.0.0.1 6399 > > info

B) get redis local profile information, maximum number of connections, timeout,tcp-keepalive

> 127.0.0.1 6399 > > config *

> 127.0.0.1 6399 > > config get maxclients

"10000"

> 127.0.0.1 6399 > > config get timeout

0

> 127.0.0.1 6399 > > config get tcp-keepalive

0

C) as can be seen from the above command, the maximum number of connections is 10000. Timeout and tcp-keepalive are not enabled, but redis-server cannot be restarted, so the configuration information can only be modified online.

> 127.0.0.1 6399 > > config set maxclients=100000

OK

> 127.0.0.1 config set timeout 6399 > > 300

OK

After doing the above measures, the number of client connections has also been disconnected.

The cause of the problem:

Why is the value obtained by 1.zabbix monitoring 0?

Execute above zabbix server

Zabbix_get-s 0.0.0.0-k 'redis [connect client,6399]'

0

The reason is that when the redis script in the agent segment grabs info information, the data that is not of numeric type is replaced with 0, so the value obtained by zabbix monitoring is 0.

two。 Why are there so many connections?

The online architecture is

Client-> VIP- > server # VIP refers to load balancing

The middle of the connection between the client and the service is forwarded through VIP, and their direct communication is the communication of layer 4 of TCP/ IP. On the VIP itself, there is a timeout setting, the default 20min, it will actively disconnect, but it does not notify the client and server, resulting in the client and server do not know that they have been disconnected. But because client's code is not set to actively disconnect, client will think it is always connected, and when it makes a request next time, VIP will tell client that the connection has been disconnected and client will re-create a new connection For the server side, that is, redis itself does not open timeout and tcp-keepalive, so the server side will not actively disconnect, so there will be more and more connections, resulting in the number of connections completely used up.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.