Shulou(Shulou.com)05/31 Report--

Xiaobian to share with you how to do Redis db connection timeout, I believe most people still do not know how to do, so share this article for everyone's reference, I hope you have a lot of harvest after reading this article, let's go to understand it together!

Problem Description:

The phenomenon is redis db access timeout.

Problem analysis:

Network traffic bottlenecks and slow access.

[machine1 ~]$ ping 192.168.10.66

PING 192.168.10.66 (192.168.10.66) 56(84) bytes of data.

64 bytes from 192.168.10.66: icmp_seq=1 ttl=64 time=17.7 ms

64 bytes from 192.168.10.66: icmp_seq=2 ttl=64 time=17.2 ms

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0

Queueing strategy: fifo

Output queue: 0/40 (size/max)

5 minute input rate 874639000 bits/sec, 85931 packets/sec -Network traffic 800m bit per second

5 minute output rate 47017000 bits/sec, 75274 packets/sec

96257027404 packets input, 21537684106526 bytes, 0 no buffer

Received 70287596 broadcasts (69442689 multicasts)

The host network card traffic has also reached a bottleneck.

Some instances of redis host memory also began to alarm, more than 90%.

After stopping the middleware application, it was found that the problem was still not solved, and the network traffic was still very high.

After switching redis db to standby, the system recovers and network traffic drops. Redis db slave library memory usage is low.

When the problem occurred, the maximum memory of the master library was 14g, but the memory of the slave library was only 1.9 g. Where does the memory footprint of the main library come from?

According to the feedback from network colleagues, the size of network requests is about tens of bytes, but the size of output reaches several M at a time.

Redis output buffers can take up a lot of memory space, much like oracle PGA memory, with a separate memory area allocated for each oracle session. Then continue to see if this case is also too much output buffer memory consumption. Looking at zabbix related metrics, we found that client_longest_output_list did increase. The output buffer takes up a large amount of memory, that is, there is a large amount of data

Output from the Redis server to certain clients.

Check the logs at that time, there are omem bigger situation.

id=411550 addr=172.24.1.30:34542 fd=280 name= age=2 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=9

omem=237928 events=rw cmd=lrange

Later, it was found that there were operations that returned more data from slowlog.

[machine1 ~]$redis-cli -p 6500 slowlog get 10

1) 1) (integer) 2245

2) (integer) 1460002216

3) (integer) 23236

4) 1) "SMEMBERS"

2) "prdkey1"

Get 9075 elements at once.

127.0.0.1:6410> scard prdkey1

(integer) 9075

After analysis with development, similar to data skew in Oracle, the data returned on a certain fragment of redis is very large, resulting in a backlog of buffers for the server to send data to the client. Subsequent development pulls this data out in batches and places it in the JVM for caching.

The above is "Redis db connection timeout how to do" all the content of this article, thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.