Shulou(Shulou.com)05/31 Report--

This article mainly introduces the example analysis of connection errors in Redis, which is very detailed and has certain reference value. Friends who are interested must finish it!

Preface

Recently, due to the increase in traffic, redis has made a series of errors, such as:

LOADING Redis is loading the dataset in memory

Use of closed network connection

Connection pool exhausted

Connection refuse by peer

Analyze them one by one.

LOADING Redis is loading the dataset in memory

There are at least two possibilities.

The available memory is too small. Modify the maxmemory in redis.conf to solve the problem.

Redis is loading the dump.rdb file at startup. Due to slow loading, redis is not available at startup.

What I encountered is the second situation. During the automatic expansion of AWS, every newly generated EC2 instance reported an error. The reason is that redis found a dump.rdb when it started, and then went to load it, causing all the services in the server to report an error, and then exited. And it takes a long time for redis to load this (I don't know why). Supervisord automatically restarts the new service and still reports an error.

Later, delete the dump.rdb file in the image before the service can start normally.

The reason for the dump.rdb file may be that there was an error in redis before, and then it was added when making the image, resulting in all the newly generated instances reporting errors.

I learned a lesson this time: stop redis and then delete dump.rdb before making an image next time.

The other three errors

At the beginning, there are all kinds of looking for information, and then various changes in configuration, resulting in these three kinds of errors appear one after another.

At first, I thought that the golang code did not handle the redis connection exception correctly, so it was useless to upgrade redigo and change the configuration of timeout, max_active, wait and so on in golang.

After tossing back and forth for about a week, I finally found something fishy in pool.Active and pool.MaxActive.

Because my MaxActive setting is 10000, so I opened 10000 go runtine to test it, found that the current number of connections pool.Active is always only about 4000, and then all kinds of errors.

At that time, the brain was also short-circuited, always thinking that redigo did not properly handle the connection of redis, which led to the failure of pool.Active to the maximum. Always thinking about changing the redigo code.

Later, there was really no way to change the ulimit. The old one was 500000 and changed to 990000. I found that the connection error was still reported, and the pool.Active still could not go up. I thought it was impossible, ah, that I thought it would be the configuration of redis itself with the maximum number of connections. Check the Internet, sure enough, redis-server has a maxclients configuration. The default is more than 4000. After changing it to 10000, the whole world is quiet.

In fact, you can't blame me, because redigo also has a max_active parameter. Who knows if redis-server has to be set up?

Redis for configuration of highly concurrent services

Redis client (that is, golang code)

Wait: true, if the connection pool is full, wait. Redis processing is very fast, and users won't feel anything after waiting for a few microseconds.

IdleTimeout: 5s can't be processed by a business logic 5s, so you should optimize your code. If set to 0, if the connection is missing, the server will not be able to retrieve it, and a zombie connection will be generated.

MaxActive: 10000 equals that the server can handle 10000 concurrent messages per second.

Redis server (i.e. redis-server)

Maxclients should be set larger than MaxActive.

Additional question: how to calculate the maximum number of files in a server?

Linux kernel-Need to "calculate" optimum ulimit and fs.file-max values according to my own server needs-Stack Overflow

This ends up being about 100 for every 1MB of ram.

For example, if it is 4 gigabytes of memory, the maximum number of open files can be set to: 4 * 1024 * 409600

The above is all the contents of the article "sample Analysis of connection errors in Redis". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.