Shulou(Shulou.com)05/31 Report--

This article introduces the relevant knowledge of "what is the use of Nginx signal set". In the operation of practical cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Nginx signal operation is the most common and very important in daily operation and maintenance. if there is a mistake in this link, it may cause business anomalies and bring losses. Therefore, it is very necessary to sort out the Nginx signal set, which can help us to deal with these tasks better.

Preface

While working, some HTTP/500 responses suddenly appeared in a ngx_lua service of a drainage test machine. Judging from the stack printed by the error log, it is due to the fact that a Lua table added in the newly released version not long ago does not exist, but is indexed by code. This is puzzling. If this is caused by a version fallback, then why is the code that uses the Lua table not fallback, but the code that defines the table?

After investigation, it is found that at that time, Nginx had just completed the hot update operation, and the old master process still existed, because in order to prepare for machine restart, the drainage traffic was cut off (but some requests were still there), and the system triggered Nginx-s stop, which caused this problem.

Scene reproduction

Now I will use a native Nginx to reproduce this process on my virtual machine where fedora26 is installed. The version of Nginx I use is 1.13.4 of the current *.

Start Nginx first:

Alex@Fedora26-64: ~ / bin_install/nginx. / sbin/nginx alex@Fedora26-64: ~ / bin_install/nginx ps auxf | grep nginx alex 6174 28876 428? Ss 14:35 0:00 nginx: master process. / sbin/nginx alex 6175 0.0 0.2 29364 2060? S 14:35 0:00\ _ nginx: worker process

You can see that both master and worker are already running.

Then we send a SIGUSR2 signal to master, and when the Nginx core receives this signal, it triggers a hot update.

Alex@Fedora26-64: ~ / bin_install/nginx kill-USR2 6174 alex@Fedora26-64: ~ / bin_install/nginx ps auxf | grep nginx alex 6174 0.0 0.1 28876 1996? Ss 14:35 0:00 nginx: master process. / sbin/nginx alex 6175 0.0 0.2 29364 2060? S 14:35 0:00\ _ nginx: worker process alex 6209 0.00.2 28876 2804? S 14:37 0:00\ _ nginx: master process. / sbin/nginx alex 6213 0.00.1 29364 2004? S 14:37 0:00\ _ nginx: worker process

You can see that the new master and the worker coming out of the master fork are already running. At this point, we then send a SIGWINCH signal to the old master. When the old master receives this signal, it sends SIGQUIT to its worker, so the worker process of the old master exits:

Alex@Fedora26-64: ~ / bin_install/nginx kill-WINCH 6174 alex@Fedora26-64: ~ / bin_install/nginx ps auxf | grep nginx alex 6174 0.0 0.1 28876 1996? Ss 14:35 0:00 nginx: master process. / sbin/nginx alex 6209 0.0 0.2 28876 2804? S 14:37 0:00\ _ nginx: master process. / sbin/nginx alex 6213 0.00.1 29364 2004? S 14:37 0:00\ _ nginx: worker process

At this point, only the old master, the new master, and the new master's worker are running, which is similar to what happened online at the time.

Then we use the stop command:

Alex@Fedora26-64: ~ / bin_install/nginx. / sbin/nginx-s stop alex@Fedora26-64: ~ / bin_install/nginx ps auxf | grep nginx alex 6174 0.00.1 28876 1996? Ss 14:35 0:00 nginx: master process. / sbin/nginx alex 6301 0.0 0.2 29364 2124? S 14:49 0:00\ _ nginx: worker process

We will find that the new master and its worker have been exited, while the old master is still running and worker is generated. This is what happened online at that time.

In fact, this phenomenon has something to do with the design of Nginx itself: before the old master is ready to generate a new master for fork, it will rename the file Nginx.pid to Nginx.pid.oldbin, and then the new master from fork will create a new Nginx.pid, which will record the pid of the new master. Nginx believes that after the hot update is complete, the mission of the old master is almost over, and then it will exit at any time, so all subsequent operations should be taken over by the new master. Of course, it doesn't work to try to hot update again by sending a SIGUSR2 to the new master without exiting the old master, and the new master will just ignore the signal and continue with its own work.

Analysis of problems

More unfortunately, the Lua table we mentioned above, the Lua file that defines it, has already been loaded into memory by LuaJIT and compiled into bytecode when running the hook of init_by_lua, so it is obvious that the old master must not have this Lua table, because the part of the Lua code it loads is the old version.

The Lua code indexing the table is not used in init_by_lua. The code is loaded in the worker process. At this time, the code in the project directory is *, so the worker process loads * code. If these worker processes process the relevant requests, there will be a Lua runtime error, and the external performance is the corresponding HTTP 500s.

After learning this lesson, we need to shut down our Nginx service more reasonably. So a more reasonable Nginx service startup and shutdown script is necessary, some scripts circulated on the Internet do not deal with this phenomenon, we should refer to the official script provided by Nginx.

Stop () {echo-n $"Stopping $prog:" killproc $prog-QUIT retval=$? Echo [$retval-eq 0] & & rm-f $lockfile return $retval}

This code is quoted from the official / etc/init.d/Nginx of Nginx [1].

Nginx signal set

Next, let's comprehensively sort out the Nginx signal set, here will not involve the details of the source code, interested students can read the relevant source code.

There are two ways to send signals to the master process, one through Nginx-s signal, and the other manually through the kill command.

The principle of this method is to generate a new process, which gets the pid of the master process from the Nginx.pid file, then sends the corresponding signal to master, and then exits. This process is called signaller.

The second approach requires us to understand the mapping of Nginx-s signal to real signals. The following table shows their mapping relationships:

Stop vs quit

Stop sends a SIGTERM signal requesting a forced exit, and quit sends a SIGQUIT to indicate a graceful exit. The specific difference is that after receiving a SIGQUIT message (note that it does not send a signal directly, so the message is used instead), the worker process closes the listening socket, closes the current idle connection (which can be preempted), and then handles all timer events in advance. Without special circumstances, you should use quit instead of stop.

Reload

After receiving the SIGHUP, the master process will perform a series of other tasks, such as parsing the configuration file, applying for shared memory, and so on, and then generate a batch of new worker processes. * send the SIGQUIT corresponding message to the old worker process, and finally realize the restart operation seamlessly.

Reopen

When the master process receives the SIGUSR1, it reopens all open files (such as logs) and sends SIGUSR1 information to each worker process, and when the worker process receives the signal, it does the same. Reopen can be used for log slicing. For example, Nginx officially provides a solution:

$mv access.log access.log.0$ kill-USR1 `cat master.nginx.pid` $sleep 1$ gzip access.log.0 # do something with access.log.0

Here sleep 1 is necessary, because there is a time window between when the master process sends a SIGUSR1 message to the worker process and the worker process actually reopens the access.log, when the worker process writes a log to the file access.log.0. Through sleep 1s, the integrity of access.log.0 log information is guaranteed (if it is compressed directly without sleep, the log is likely to be lost).

Hot update

Sometimes we need to do a binary hot update, which is included in Nginx's design, but it can't be done through the command line provided by Nginx. We need to send the signal manually.

Through the reproduction of the above problems, you should already know how to hot update, we first need to send SIGUSR2 to the current master process, then master will rename Nginx.pid to Nginx.pid.oldbin, and then fork a new process, the new process will replace the current process image with the new NginxELF file through the system call execve to become the new master process. After the new master process is up, operations such as configuration file parsing are performed, and then the new worker process is fork out and started to work.

Then we send a SIGWINCH signal to the old master, and then the old master process sends SIGQUIT information to its worker process, causing the worker process to exit. Sending both SIGWINCH and SIGQUIT to the master process exits the worker process, but the former does not cause the master process to exit as well.

* if we feel that the mission of the old master process is complete, we can send it a SIGQUIT signal to exit.

How the worker process handles signal messages from master

In fact, instead of using the kill function, the master process communicates with the worker process by using the Nginxchannel,master process implemented through the pipeline to write information (such as signal information) to one end of the pipeline, while the worker process receives information from the other end. Nginxchannel events are added to the event scheduler (such as epoll,kqueue) as soon as the worker process is up, so when data is sent from master, it can be notified by the event scheduler.

Many people might think that when the master process sends information to the worker process, the worker process will immediately respond to the corresponding action, but the worker process is very busy, it constantly handles network events and timer events, and when the handler of the Nginxchannel event is called, the Nginx only handles a few flag bits. These actions are actually performed after a round of event scheduling is completed. So there is a time window, especially when the business is complex and the traffic is huge, this window may be enlarged, which is why sleep 1s is required in the log slicing solution provided by Nginx.

Of course, we can also bypass the master process and send signals directly to the worker process. The signals that worker can handle are

This is the end of the content of "what is the use of Nginx signal set". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.