Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you about how to decode the most neglected CPU and high memory consumption of Redis. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

When we use Redis, we always encounter some problems of high CPU and high memory footprint on redis-server. Let's take a few real-world cases as examples to discuss several situations that are easy to ignore when using Redis.

1. Short connections lead to high CPU

A user reported that the QPS is not high, from the monitoring, the CPU is indeed on the high side. Since the QPS is not high, redis-server itself is likely to be doing some cleaning work or users are executing commands with high complexity. After troubleshooting, there is no expired deletion of key, and no commands with high complexity are executed.

After the perf analysis of redis-server on the machine, it is found that the function listSearchKey occupies a relatively high CPU. The analysis call stack finds that listSearchKey is frequently called when the connection is released, and the user reports that the short connection is used, so it is inferred that the frequent release of the connection leads to an increase in CPU usage.

1. Contrast examples

The following uses the redis-benchmark tool to do a comparative experiment using long connection and short connection respectively. Redis-server is community version 4.0.10.

1) long connection test

Send 50w ping commands to redis-server using 10000 persistent connections:

. / redis-benchmark-h host-p port-t ping-c 10000-n 500000-k 1

Final QPS:

PING_INLINE: 92902.27 requests per secondPING_BULK: 93580.38 requests per second

According to the analysis of redis-server, it is found that readQueryFromClient occupies the highest CPU, that is, it is mainly processing requests from the user side.

2) short connection test

Send 50w ping commands to redis-server using 10000 short connections:

. / redis-benchmark-h host-p port-t ping-c 10000-n 500000-k 0

Final QPS:

PING_INLINE: 15187.18 requests per secondPING_BULK: 16471.75 requests per second

According to the analysis of redis-server, it is found that listSearchKey occupies the highest CPU, while the proportion of readQueryFromClient in CPU is much lower than that of listSearchKey, that is to say, CPU is a little "out of business", processing user requests has become a sideline, while searching list has become the main business. Therefore, under the same amount of business requests, the use of short connections will increase the burden of CPU.

From the perspective of QPS, there is a large gap between short connection and long connection because of two reasons:

The network overhead introduced each time the connection is reestablished.

When a connection is released, redis-server consumes additional CPU cycles to clean up. (you can try to optimize this from the redis-server side)

2. Redis connection release

From the code level, let's look at what redis-server will do after the client initiates the release of the connection. When the redis-server receives the disconnection request from the client, it will go directly to the freeClient.

Void freeClient (client * c) {listNode * ln; / *. * / / * Free the query buffer * / sdsfree (c-> querybuf); sdsfree (c-> pending_querybuf); c-> querybuf = NULL; / * Deallocate structures used to block on blocking ops. * / if (c-> flags & CLIENT_BLOCKED) unblockClient (c); dictRelease (c-> bpop.keys); / * UNWATCH all the keys * / unwatchAllKeys (c); listRelease (c-> watched_keys); / * Unsubscribe from all the pubsub channels * / pubsubUnsubscribeAllChannels (c Magol 0); pubsubUnsubscribeAllPatterns (c-Magol 0); dictRelease (c-> pubsub_channels); listRelease (c-> pubsub_patterns); / * Free data structures. * / listRelease (c-> reply); freeClientArgv (c); / * Unlink the client: this will close the socket, remove the I * handlers, and remove references of the client from different * places where active clients may be referenced. * / / * redis-server maintains a server.clients linked list. When the client establishes a connection, create a new client object and append it to the server.clients. When the connection is released, the client object * / unlinkClient (c) needs to be deleted from the server.clients; / *. /} void unlinkClient (client * c) {listNode * ln; / * If this is marked as current client unset it. * / if (server.current_client = = c) server.current_client = NULL; / * Certain operations must be done only if the client has an active socket. * If the client was already unlinked or if it's a "fake client" the * fd is already set to-1. * / if (c-> fd! =-1) {/ * search the server.clients linked list, and then delete the client node object. Here the complexity is O (N) * / ln = listSearchKey (server.clients,c); serverAssert (ln! = NULL); listDelNode (server.clients,ln); / * Unregister async Ipool O handlers and close the socket. * / aeDeleteFileEvent (server.el,c- > fd,AE_READABLE); aeDeleteFileEvent (server.el,c- > fd,AE_WRITABLE); close (c-> fd); c-> fd =-1;} / *. , /

So every time the connection is disconnected, there is an O (N) operation. For in-memory databases like redis, we should avoid O (N) operations as much as possible, especially in scenarios with a large number of connections, which has an obvious impact on performance. Although users can avoid it as long as they do not use short connections, in real-world scenarios, users may also establish some short connections after the client connection pool is full.

3. Optimization

From the above analysis, the O (N) operation is performed every time the connection is released, so can the complexity be reduced to O (1)?

The problem is very simple. Server.clients is a two-way linked list, and as long as the client object remembers its memory address when it is created, there is no need to traverse the server.clients when it is released. Next, try to optimize:

Client * createClient (int fd) {client * c = zmalloc (sizeof (client)); / * * / listSetFreeMethod (c-> pubsub_patterns,decrRefCountVoid); listSetMatchMethod (c-> pubsub_patterns,listMatchObjects); if (fd! =-1) {/ * the listNode address of the list where the client records itself * / c-> client_list_node = listAddNodeTailEx (server.clients,c);} initClientMultiState (c); return c;} void unlinkClient (client * c) {listNode * ln; / * If this is marked as current client unset it. * / if (server.current_client = = c) server.current_client = NULL; / * Certain operations must be done only if the client has an active socket. * If the client was already unlinked or if it's a "fake client" the * fd is already set to-1. * / if (c-> fd! =-1) {/ * there is no need to search the server.clients linked list * / ln = listSearchKey (server.clients,c); / / serverAssert (ln! = NULL); / / listDelNode (server.clients,ln); listDelNode (server.clients,c-> client_list_node); / * Unregister async Ipool O handlers and close the socket. * / aeDeleteFileEvent (server.el,c- > fd,AE_READABLE); aeDeleteFileEvent (server.el,c- > fd,AE_WRITABLE); close (c-> fd); c-> fd =-1;} / *. , /

Optimized short connection test

Send 50w ping commands to redis-server using 10000 short connections:

. / redis-benchmark-h host-p port-t ping-c 10000-n 500000-k 0

Final QPS:

PING_INLINE: 21884.23 requests per secondPING_BULK: 21454.62 requests per second

Compared with that before optimization, the performance of short connections can be improved by 30%, so it can ensure that the performance will not be too bad when there are short connections.

Second, the info command leads to high CPU.

Some users monitor the status of redis by executing info commands on a regular basis, which can lead to high CPU usage to some extent. When info is executed frequently, through perf analysis, it is found that the functions of getClientsMaxBuffers, getClientOutputBufferMemoryUsage and getMemoryOverheadData occupy more CPU.

Through the Info command, you can pull some of the following status information (not all listed) to the redis-server side:

The longest outputbuffer list length client_biggest_input_buf:0 on clientconnected_clients:1client_longest_output_list:0 / / redis-server. / / longest inputbuffer byte length on redis-server blocked_clients:0Memoryused_memory:848392used_memory_human:828.51Kused_memory_rss:3620864used_memory_rss_human:3.45Mused_memory_peak:619108296used_memory_peak_human:590.43Mused_memory_peak_perc:0.14%used_memory_overhead:836182 / / except dataset The amount of memory used by redis-server to maintain its own structure used_memory_startup:786552used_memory_dataset:12210used_memory_dataset_perc:19.74% in order to get the status of client_longest_output_list and client_longest_output_list, you need to traverse all the client on the redis-server side, as shown in getClientsMaxBuffers. You may see that the same O (N) operation exists here. Void getClientsMaxBuffers (unsigned long * longest_output_list, unsigned long * biggest_input_buffer) {client * c; listNode * ln; listIter li; unsigned long lol = 0, bib = 0; / * traverse all client with complexity O (N) * / listRewind (server.clients,&li); while ((ln = listNext (& li))! = NULL) {c = listNodeValue (ln) If (listLength (c-> reply) > lol) lol = listLength (c-> reply); if (sdslen (c-> querybuf) > bib) bib = sdslen (c-> querybuf);} * longest_output_list = lol; * biggest_input_buffer = bib } in order to get the used_memory_overhead status, you also need to traverse all the client to calculate the total amount of memory occupied by the outputBuffer of all client, as shown in getMemoryOverheadData: struct redisMemOverhead * getMemoryOverheadData (void) {/ *. * / mem = 0; if (server.repl_backlog) mem + = zmalloc_size (server.repl_backlog); mh- > repl_backlog = mem; mem_total + = mem; / *. * / mem = 0; if (listLength (server.clients)) {listIter li; listNode * ln / * iterate through all client and calculate the sum of memory occupied by all client outputBuffer with a complexity of O (N) * / listRewind (server.clients,&li); while ((ln = listNext (& li) {client * c = listNodeValue (ln); if (c-> flags & CLIENT_SLAVE) continue; mem + = getClientOutputBufferMemoryUsage (c) Mem + = sdsAllocSize (c-> querybuf); mem + = sizeof (client);}} mh- > clients_normal = mem; mem_total+=mem; mem = 0; if (server.aof_state! = AOF_OFF) {mem + = sdslen (server.aof_buf); mem + = aofRewriteBufferSize ();} mh- > aof_buffer = mem; mem_total+=mem; / *. * / return mh;} experiment

From the above analysis, we know that when the number of connections is high (N of O (N) is large), if the info command is executed frequently, it will take up more CPU.

1) establish a connection and keep executing info commands

Func main () {c, err: = redis.Dial ("tcp" Addr) if err! = nil { Fmt.Println ("Connect to redis error:" Err) return} For { C.Do ("info")} Return }

The experimental results show that the occupancy of CPU is only about 20%.

2) establish 9999 idle connections, and one connection continuously executes info

Func main () {clients: = [] redis.Conn {} for I: = 0; I

< 9999; i++ { c, err := redis.Dial("tcp", addr) if err != nil { fmt.Println("Connect to redis error:", err) return } clients = append(clients, c) } c, err := redis.Dial("tcp", addr) if err != nil { fmt.Println("Connect to redis error:", err) return } for { _, err = c.Do("info") if err != nil { panic(err) } } return } 实验结果表明CPU能够达到80%，所以在连接数较高时，尽量避免使用info命令。 3）pipeline导致内存占用高 有用户发现在使用pipeline做只读操作时，redis-server的内存容量偶尔也会出现明显的上涨, 这是对pipeline的使不当造成的。下面先以一个简单的例子来说明Redis的pipeline逻辑是怎样的。 下面通过golang语言实现以pipeline的方式从redis-server端读取key1、key2、key3。 import ( "fmt" "github.com/garyburd/redigo/redis")func main(){ c, err := redis.Dial("tcp", "127.0.0.1:6379") if err != nil { panic(err) } c.Send("get", "key1") //缓存到client端的buffer中 c.Send("get", "key2") //缓存到client端的buffer中 c.Send("get", "key3") //缓存到client端的buffer中 c.Flush() //将buffer中的内容以一特定的协议格式发送到redis-server端 fmt.Println(redis.String(c.Receive())) fmt.Println(redis.String(c.Receive())) fmt.Println(redis.String(c.Receive()))} 而此时server端收到的内容为： *2 $3 get $4 key1 *2 $3 get $4 key2 *2 $3 get $4 key3 下面是一段redis-server端非正式的代码处理逻辑，redis-server端从接收到的内容依次解析出命令、执行命令、将执行结果缓存到replyBuffer中，并将用户端标记为有内容需要写出。等到下次事件调度时再将replyBuffer中的内容通过socket发送到client，所以并不是处理完一条命令就将结果返回用户端。 readQueryFromClient(client* c) { read(c->

Querybuf) / c-> query= "* 2 $3 get $4 key1 * 2 $3 get $4 key2 * 2 $3 get $4 key3" cmdsNum = parseCmdNum (c-> querybuf) / / cmdNum = 3 while (cmsNum--) {cmd = parseCmd (c-> querybuf) / / cmd: get key1, get key2, get key3 reply = execCmd (cmd) appendReplyBuffer (reply) markClientPendingWrite (c)}}

Consider such a situation:

If the client program is slow to read the contents from the receiving buffer of TCP through c.Receive (), or c.Receive () is not executed because of some BUG, when the receiving buffer is full, the TCP sliding window on the server side is 0, causing the server side to not be able to send the content in the replyBuffer, so the server takes up extra memory due to the delayed release of the server. The problem becomes more acute when pipeline packages too many commands at a time and contains commands such as mget, hgetall, lrange, and so on, that manipulate multiple objects.

The above situations are very simple problems. Without complex logic, they are not a problem in most scenarios, but developers still need to pay attention to these details in order to make good use of Redis in some extreme scenarios. Recommendations:

Try not to use short connections

Try not to use info frequently in scenarios with a high number of connections.

When using pipeline, you should receive the result of request processing in a timely manner, and pipeline should not package too many requests at once.

The above is the problem of how to decode the most neglected CPU and high memory footprint of Redis shared by Xiaobian. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.