Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to analyze the Redis timeout. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Redis plays a more and more important role in distributed applications. With tens of thousands of lines of code, it implements a high-performance data storage service. Recently, there have been several redis timeouts in the cm8 cluster in the dump center, but looking at the relevant memory of redis machines, we did not find that there was not enough memory, or the memory was swapped. After looking at the redis source code, we found that redis would time out in some cases. The details are as follows.

1. The Internet. The processing of Redis is closely related to the network. If there is a flash break in the network, the redis timeout is easy to occur. If this happens, you should first check the bandwidth information of the redis machine network to determine whether a flash break has occurred.

two。 Memory. All the data in redis is stored in memory. When there is not enough physical memory, linux os will use swap memory, resulting in memory swapping. At this time, if there is a redis call command, it will cause redis timeout. Here, you can adjust the / proc/sys/vm/swappiness parameter to set how much physical memory will be used for swap.

Int rdbSaveBackground (char * filename) {pid_t childpid; long long start; if (server.rdb_child_pid! =-1) return REDIS_ERR; serverserver.dirty_before_bgsave = server.dirty; server.lastbgsave_try = time (NULL); start = ustime (); if ((childpid = fork ()) = 0) {int retval; / * Child * / if (server.ipfd > 0) close (server.ipfd) If (server.sofd > 0) close (server.sofd); retval = rdbSave (filename); if (retval = = REDIS_OK) {size_t private_dirty = zmalloc_get_private_dirty () If (private_dirty) {redisLog (REDIS_NOTICE, "RDB:% zu MB of memory used by copy-on-write", private_dirty/ (1024: 1024));}} exitFromChild ((retval = = REDIS_OK)? 0: 1) } else {/ * Parent * / server.stat_fork_time = ustime ()-start; if (childpid = =-1) {server.lastbgsave_status = REDIS_ERR; redisLog (REDIS_WARNING, "Can't save in background: fork:% s", strerror (errno)); return REDIS_ERR } redisLog (REDIS_NOTICE, "Background saving started by pid% d", childpid); server.rdb_save_time_start = time (NULL); server.rdb_child_pid = childpid; updateDictResizePolicy (); return REDIS_OK;} return REDIS_OK; / * unreached * /}

Program 1

In addition, there are some special circumstances that can lead to swap. When we use rdb for redis cluster persistence, we may run out of physical memory (aof persistence just keeps supporting the continuous addition of redis cluster change operations, which is less likely to cause swap). When using rdb persistence, as shown in Program 1, the main process fork a child process to dump redis all the data, and the main process still serves the client. At this time, the main process and child process share the same memory area, and the linux kernel uses write-time replication to ensure the security of data. In this mode, if the client sends a write request, the kernel assigns the page to a new page and marks it as write, before writing the write request to the page. Therefore, in the case of rdb persistence, if there are other requests, redis will use more memory and swap is more likely to occur. Therefore, using rdb persistence as little as possible in scenarios that can be quickly recovered can make the conditions of rdb dump a little more stringent. Of course, you can choose aof, but aof also has its own shortcomings. In addition, the master-slave structure after 2.6can also be used to separate read and write, so that there is no read-write scenario on the server process. Redis single-process processing commands. Redis supports both udp and tcp connections. The redis client sends information including the redis command to the redis server. After receiving the information, the redis server parses the command and performs the corresponding operation. The specific flow of the redis processing command is as follows. First, the server establishes a connection as shown in Program 2, and returns the file descriptor after creating the socket,bind,listen:

Server.ipfd = anetTcpServer (server.neterr,server.port,server.bindaddr)

Program 2

For a service like redis, it needs to handle thousands of connections (up to 655350) and uses multiplexing to handle multiple connections. Here redis provides epoll,select and kqueue to implement, and here epoll (ae.c) is used by default. After getting the file descriptor fd returned by the listen function, redis adds fd and its handling acceptTcpHandler function to the event-driven linked list. In fact, when joining the event queue, the program 4 event driver adds socket-related fd file descriptors to the listening events of the epoll.

If (server.ipfd > 0 & aeCreateFileEvent (server.el,server.ipfd,AE_READABLE, acceptTcpHandler,NULL) = = AE_ERR) redisPanic ("Unrecoverable error creating server.ipfd file event."); int aeCreateFileEvent (aeEventLoop * eventLoop, int fd, int mask, aeFileProc * proc, void * clientData) {if (fd > = eventLoop- > setsize) {errno = ERANGE; return AE_ERR;} aeFileEvent * fe = & eventLoop- > events [fd] If (aeApiAddEvent (eventLoop, fd, mask) =-1) return AE_ERR; fe- > mask | = mask; if (mask & AE_READABLE) fe- > rfileProc = proc; if (mask & AE_WRITABLE) fe- > wfileProc = proc; fe- > clientDataclientData = clientData; if (fd > eventLoop- > maxfd) eventLoop- > maxfd = fd; return AE_OK;}

Program 3

Static int aeApiAddEvent (aeEventLoop * eventLoop, int fd, int mask) {aeApiState * state = eventLoop- > apidata; struct epoll_event ee; / * If the fd was already monitored for some event, we need a MOD * operation. Otherwise we need an ADD operation. * / int op = eventLoop- > events [FD] .mask = = AE_NONE? EPOLL_CTL_ADD: EPOLL_CTL_MOD; ee.events = 0; mask | = eventLoop- > events [FD] .mask; / * Merge old events * / if (mask & AE_READABLE) ee.events | = EPOLLIN; if (mask & AE_WRITABLE) ee.events | = EPOLLOUT; ee.data.u64 = 0; / * avoid valgrind warning * / ee.data.fd = fd If (epoll_ctl (state- > epfd,op,fd,&ee) =-1) return-1; return 0;}

Program 4

After all the event drivers are finished in the initial session, as shown in program 5, the main process obtains the io-ready file descriptor and its corresponding handler based on numevents = aeApiPoll (eventLoop, tvp), and processes the fd. The general flow is accept ()-> createclient ()-> readQueryFromClient (). ReadQueryFromClient () reads the redis command in the information-> processInputBuffer ()-> call () * * to complete the command.

Void aeMain (aeEventLoop * eventLoop) {eventLoop- > stop = 0; while (! eventLoop- > stop) {if (eventLoop- > beforesleep! = NULL) eventLoop- > beforesleep (eventLoop); aeProcessEvents (eventLoop, AE_ALL_EVENTS);} int aeProcessEvents (aeEventLoop * eventLoop, int flags) {--numevents = aeApiPoll (eventLoop, tvp) For (j = 0; j)

Events [eventLoop-> fired.fd]; int mask = eventLoop- > fired.mask; int fd = eventLoop- > fired.fd; int rfired = 0; / * note the fe- > mask & mask &. Code: maybe an already processed * event removed an element that fired and we still didn't * processed, so we check if the event is still valid. * / if (fe- > mask & mask & AE_READABLE) {rfired = 1; fe- > rfileProc (eventLoop,fd,fe- > clientData,mask);} if (fe- > mask & mask & AE_WRITABLE) {if (! rfired | | fe- > wfileProc! = fe- > rfileProc) fe- > wfileProc (eventLoop,fd,fe- > clientData,mask) } processed++;}}

Program 5

From the above code, we can see that redis uses ae event-driven combined with epoll multiplexing to achieve serial command processing. Therefore, some slow commands such as sort,hgetall,union,mget will make the processing time of a single command longer, which can easily lead to the subsequent command time out. Therefore, we need to avoid using slow commands as far as possible, such as changing the hash format to kv self-parsing, and second, increasing the number of redis instances, and each redis server calls as few slow commands as possible.

The above is the editor for you to share how to analyze the Redis timeout, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.