Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to understand the nginx signal set". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to understand the nginx signal set.

Preface

While working, some HTTP/500 responses suddenly appeared in a ngx_lua service of a drainage test machine. Judging from the stack printed by the error log, it is due to the fact that a Lua table added in the newly released version not long ago does not exist, but is indexed by code. This is puzzling. If this is caused by a version fallback, then why is the code that uses the Lua table not fallback, but the code that defines the table?

After investigation, it is found that at that time, nginx had just completed the hot update operation, and the old master process still existed, because in order to prepare for machine restart, the drainage traffic was cut off (but some requests were still there), and the system triggered nginx-s stop, which caused this problem.

Scene reproduction

Now I will use a native nginx to reproduce this process on my virtual machine with fedora26 installed. The version of nginx I am using is the latest 1.13.4.

Start nginx first

You can see that both master and worker are already running.

Then we send a SIGUSR2 signal to master, and when the nginx core receives this signal, it triggers a hot update.

You can see that the new master and the worker coming out of the master fork are already running. At this point, we then send a SIGWINCH signal to the old master. When the old master receives this signal, it sends SIGQUIT to its worker, so the worker process of the old master exits:

At this point, only the old master, the new master, and the new master's worker are running, which is similar to what happened online at the time.

Then we use the stop command:

We will find that the new master and its worker have been exited, while the old master is still running and worker is generated. This is what happened online at that time.

In fact, this phenomenon has something to do with the design of nginx itself: before the old master is ready to generate a new master for fork, it will rename the file nginx.pid to nginx.pid.oldbin, and then the new master from fork will create a new nginx.pid, which will record the pid of the new master. Nginx believes that after the hot update is complete, the mission of the old master is almost over, and then it will exit at any time, so all subsequent operations should be taken over by the new master. Of course, it doesn't work to try to hot update again by sending a SIGUSR2 to the new master without exiting the old master, and the new master will just ignore the signal and continue with its own work.

Analysis of problems

More unfortunately, the Lua table we mentioned above, the Lua file that defines it, has already been loaded into memory by LuaJIT and compiled into bytecode when running the hook of init_by_lua, so it is obvious that the old master must not have this Lua table, because the part of the Lua code it loads is the old version.

The Lua code indexing the table is not used in init_by_lua, these codes are loaded in the worker process, at this time the code in the project directory is up-to-date, so the worker process loads the latest code, if these worker processes deal with related requests, there will be Lua runtime errors, and the external performance is the corresponding HTTP 500.

After learning this lesson, we need to shut down our nginx service more reasonably. So a more reasonable nginx service startup and shutdown script is necessary, some scripts circulated on the Internet do not deal with this phenomenon, we should refer to the official script provided by NGINX.

This code is quoted from the official / etc/init.d/nginx of NGINX.

Nginx signal set

Next, let's comprehensively sort out the nginx signal set, here will not involve the details of the source code, interested students can read the relevant source code.

There are two ways to send signals to the master process, one through the nginx-s signal, and the other manually through the kill command.

The principle of the first way is to generate a new process, which gets the pid of the master process from the nginx.pid file, then sends the corresponding signal to master, and then exits, which is called signaller.

The second approach requires us to understand the mapping of nginx-s signal to real signals. The following table shows their mapping relationships:

Operation signal reload SIGHUP reopen SIGUSR1 stop SIGTERM quit SIGQUIT hot update SIGUSR2 & SIGWINCH & SIGQUIT stop vs quit

Stop sends a SIGTERM signal requesting a forced exit, and quit sends a SIGQUIT to indicate a graceful exit. The specific difference is that after receiving a SIGQUIT message (note that it does not send a signal directly, so the message is used instead), the worker process closes the listening socket, closes the currently idle connection (which can be preempted), then handles all timer events in advance, and finally exits. Without special circumstances, you should use quit instead of stop.

Reload

After receiving the SIGHUP, the master process will do a series of other tasks such as configuration file parsing, shared memory request, and so on, and then generate a batch of new worker processes, and finally send the SIGQUIT corresponding message to the old worker process, and finally realize the restart operation seamlessly.

Reopen

When the master process receives the SIGUSR1, it reopens all open files (such as logs) and sends SIGUSR1 information to each worker process, and when the worker process receives the signal, it does the same. Reopen can be used for log slicing. For example, NGINX officially provides a solution:

Here sleep 1 is necessary, because there is a time window between when the master process sends a SIGUSR1 message to the worker process and the worker process actually reopens the access.log, when the worker process writes a log to the file access.log.0. Through sleep 1s, the integrity of access.log.0 log information is guaranteed (if it is compressed directly without sleep, the log is likely to be lost).

Hot update

Sometimes we need to do a binary hot update, which is included in nginx's design, but it can't be done through the command line provided by nginx. We need to send the signal manually.

Through the reproduction of the above problems, you should already know how to hot update, we first need to send SIGUSR2 to the current master process, then master will rename nginx.pid to nginx.pid.oldbin, and then fork a new process, the new process will replace the current process image with the new nginx ELF file through the system call execve to become the new master process. After the new master process is up, operations such as configuration file parsing are performed, and then the new worker process is fork out and started to work.

Then we send a SIGWINCH signal to the old master, and then the old master process sends SIGQUIT information to its worker process, causing the worker process to exit. Sending both SIGWINCH and SIGQUIT to the master process exits the worker process, but the former does not cause the master process to exit as well.

Finally, if we feel that the mission of the old master process is complete, we can send it a SIGQUIT signal to exit.

How the worker process handles signal messages from master

In fact, instead of using the kill function, the master process communicates with the worker process by using the nginx channel,master process implemented through the pipeline to write information (such as signal information) to one end of the pipeline, while the worker process receives information from the other end. Nginx channel events are added to the event scheduler (such as epoll,kqueue) as soon as the worker process is up, so when data is sent from master, it can be notified by the event scheduler.

Nginx is designed for a reason. As an excellent reverse proxy server, nginx pursues the ultimate high performance, while signal handler interrupts the operation of the worker process, making all events paused for a time window, which has a certain loss of performance.

Many people might think that when the master process sends information to the worker process, the worker process will immediately respond to the corresponding action, but the worker process is very busy, it constantly handles network events and timer events, and when the handler of the nginx channel event is called, the nginx only handles a few flag bits. These actions are actually performed after a round of event scheduling is completed. So there is a time window, especially when the business is complex and the traffic is huge, this window may be enlarged, which is why sleep 1s is required in the log slicing solution provided by NGINX.

Of course, we can also bypass the master process and send signals directly to the worker process. The signals that worker can handle are

Signal effect SIGINT forced exit SIGTERM forced exit SIGQUIT elegant exit SIGUSR1 reopen the file

At this point, I believe you have a deeper understanding of "how to understand the nginx signal set". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.