Shulou(Shulou.com)11/24 Report--

This article comes from the official account of Wechat: developing Internal skills practice (ID:kfngxl). Author: Zhang Yanfei allen

Hello, everyone. I'm Brother Fei!

In a single-process network programming model. All network-related actions are done in one process, such as listening to the creation of socket, bind, listen. Another example is the creation of epoll, the addition of listening events, and the occurrence of epoll_wait waiting time. It's all done in one process.

The interaction between a client and a server using epoll is shown in the following figure.

The following is a general code example (students who don't have the patience to watch can go first).

Int main () / listen lfd = socket (AF_INET,SOCK_STREAM,0); bind (lfd,) listen (lfd,); / / create an epoll object and manage the events of listen socket efd = epoll_create (); epoll_ctl (efd, EPOLL_CTL_ADD, lfd,); / / event loop for () {size_t nready = epoll_wait (efd, ep,); for (int I = 0; I

< nready; ++i){ if(ep[i].data.fd == lfd){ //lfd上发生事件表示都连接到达，accept接收它 fd = accept(listenfd, ); epoll_ctl(efd, EPOLL_CTL_ADD, fd, ); }else{ //其它socket发生的事件都是读写请求、或者关闭连接 } } }}在单进程模型中，不管有多少的连接，是几万还是几十万，服务器都是通过 epoll 来监控这些连接 socket 上的可读和可写事件。当某个 socket 上有数据发生的时候，再以非阻塞的方式对 socket 进行读写操作。 事实上，Redis 5.0 及以前的版本中，它的网络部分去掉对 handler 的封装，去掉时间事件以后，代码基本和上述 demo 非常接近。而且因为 Redis 的业务特点只需要内存 IO，且 CPU 计算少，所以可以达到数万的 QPS。 但是单进程的问题也是显而易见的，没有办法充分发挥多核的优势。所以目前业界绝大部分的后端服务还都是需要基于多进程的方式来进行开发的。到了多进程的时候，更复杂的问题多进程之间的配合和协作问题就产生了。比如 哪个进程执行监听 listen ，以及 accept 接收新连接？ 哪个进程负责发现用户连接上的读写事件？ 当有用户请求到达的时候，如何均匀地将请求分散到不同的进程中？ 需不需要单独搞一部分进程执行计算工作 ... 事实上，以上这些问题并没有标准答案。各大应用或者网络框架都有自己不同的实现方式。为此业界还专门总结出了两类网络设计模式 - Reactor 和 Proactor。不过今天我不想讨论这种抽象模式，而是想带大家看一个具体的 Case - Nginx 是如何在多进程下使用 epoll 的。 1、 Nginx Master 进程初始化在 Nginx 中，将进程分成了两类。一类是 Master 进程，一类是 Worker 进程。 在 Master 进程中，主要的任务是负责启动整个程序、读取配置文件、监听和处理各种信号，并对 Worker 进程进行统筹管理。 不过今天我们要查看的重点问题是看网络。在 Master 进程中，和网络相关的操作非常简单就是创建了 socket 并对其进行 bind 和 监听。 具体细节我们来看 Main 函数。 //file: src/core/nginx.cint ngx_cdecl main(int argc, char *const *argv){ ngx_cycle_t *cycle, init_cycle; //1.1 ngx_init_cycle 中开启监听 cycle = ngx_init_cycle(&init_cycle); //1.2 启动主进程循环 ngx_master_process_cycle(cycle);}在 Nginx 中，ngx_cycle_t 是非常核心的一个结构体。这个结构体存储了很多东西，也贯穿了好多的函数。其中对端口的 bind 和 listen 就是在它执行时完成的。 ngx_master_process_cycle 是 Master 进程的主事件循环。它先是根据配置启动指定数量的 Worker 进程，然后就开始关注和处理重启、退出等信号。接下来我们分两个小节来更详细地看。 1.1 Nginx 的服务端口监听我们看下 ngx_init_cycle 中是如何执行 bind 和 listen 的。 //file: src/core/ngx_cycle.cngx_cycle_t *ngx_init_cycle(ngx_cycle_t *old_cycle){ if (ngx_open_listening_sockets(cycle) != NGX_OK) { goto failed; }}真正的监听还是在 ngx_open_listening_sockets 函数中，继续看它的源码。 //file: src/core/ngx_connection.cngx_int_t ngx_open_listening_sockets(ngx_cycle_t *cycle){ //要监听的 socket 对象 ls = cycle->

Listening.elts; for (I = 0; I)

< cycle->

Listening.nelts; iTunes +) {/ / get the I th socket s = ngx_socket (LS [I] .sockaddr-> sa_family, LS [I] .type, 0); / bind bind (s, LS [I] .sockaddr, ls [I] .socklen) / / listen listen (s, LS [I] .backlog) LS [I] .backlog) LS [I] .fd = s;} in this function, iterate over the socket to be listened on. If REUSEPORT configuration is enabled, set socket to the SO_REUSEPORT option first. Then there are the familiar bind and listen. Therefore, bind and listen are done in the Master process.

The main loop of the Master process does two main things in ngx_master_process_cycle.

Start the Worker process

Push the Master process into the event loop

When the Worker process is created, the Worker process completely copies its own resources, including the socket handle of the listen state, through the fork system call.

Let's look at the detailed code next.

/ / file: src/os/unix/ngx_process_cycle.cvoid ngx_master_process_cycle (ngx_cycle_t * cycle) {ngx_start_worker_processes (cycle, ccf-worker_processes, NGX_PROCESS_RESPAWN); / / enter the main loop and wait to receive various signals for () {/ / ngx_quit / / ngx_reconfigure / / ngx_restart}} the main process reads the number of Worker processes ccf- > worker_processes in the configuration. Use ngx_start_worker_processes to start a specified number of Worker.

/ / file:src/os/unix/ngx_process_cycle.cstatic void ngx_start_worker_processes () {for I = 0; in; iTunes + {ngx_spawn_process (cycle, ngx_worker_process_cycle, (void *) (intptr_t) I, "worker process", type);}} it is worth noting in the above code that there are several parameters when calling ngx_spawn_process

Core data structure of cycle:nginx

Entry function of ngx_worker_process_cycle:worker process

I: serial number of the current worker

/ / file: src/os/unix/ngx_process.cngx_pid_t ngx_spawn_process (ngx_cycle_t * cycle, ngx_spawn_proc_pt proc,) {pid = fork (); switch (pid) {case-1: / / error case 0: / / Child process created successfully ngx_parent = ngx_pid; ngx_pid = ngx_getpid (); proc (cycle, data); break; default: break }} call fork in ngx_spawn_process to create the process, and after the creation is successful, the Worker process will enter the ngx_worker_process_cycle for processing.

Summary: on the network, the master process is actually just a listen. The socket after listen is stored in cycle- > listening. The rest of the network operation is done in Worker.

2. Worker process processing as seen in the above section, the Master process actually does not do much about the network, just bind and listen. There are no epoll-related function calls, let alone accept receive connections, as well as read and write functions. Then all these details must have been completed in the Worker process.

Indeed, epoll_create, epoll_ctl, and epoll_wait are all executed in the Worker process.

In the Worker process, an epoll kernel object is created, registers the events it wants to listen to through epoll_ctl, and then calls epoll_wait to enter the event loop.

/ / file: src/os/unix/ngx_process_cycle.cstatic void ngx_worker_process_cycle (ngx_cycle_t * cycle, void * data) {/ / 2.2 the Worker process initializes the compiled modules ngx_worker_process_init (cycle, worker); / / enters the event loop for () {/ / 2.3 enters epollwait ngx_process_events_and_timers (cycle);}} Let's take a closer look.

2.1 Nginx's network-related module, leaving aside the workflow of Worker, let's first learn about a background-Nginx module.

Nginx adopts a modular architecture, and its modules include core modules, standard HTTP modules, optional HTTP modules, mail service modules and third-party modules. Each module exists in the form of a module, corresponding to a ngx_module_s structure. It is a very excellent software architecture to realize software pluggability in this way.

Each module implements a variety of init_xxx and exit_xxx methods according to its own needs for Nginx to call at the right time.

/ / file: src/core/ngx_module.hstruct ngx_module_s {. Ngx_uint_t version; void * ctx; ngx_command_t * commands; ngx_uint_t type; ngx_int_t (* init_master) (ngx_log_t * log); ngx_int_t (* init_module) (ngx_cycle_t * cycle); ngx_int_t (* init_process) (ngx_cycle_t * cycle) Ngx_int_t (* init_thread) (ngx_cycle_t * cycle); void (* exit_thread) (ngx_cycle_t * cycle); void (* exit_process) (ngx_cycle_t * cycle); void (* exit_master) (ngx_cycle_t * cycle);.} Among them, the module related to the network are ngx_events_module, ngx_event_core_module and specific network underlying modules such as ngx_epoll_module, ngx_kqueue_module and so on.

For ngx_epoll_module, it defines various actions methods (add events, delete events, add connections, and so on) in its context ngx_epoll_module_ctx.

/ / file:src/event/ngx_event.htypedef struct {ngx_str_t * name; void * (* create_conf) (ngx_cycle_t * cycle); char * (* init_conf) (ngx_cycle_t * cycle, void * conf); ngx_event_actions_t actions;} ngx_event_module_t / / file:src/event/modules/ngx_epoll_module.cstatic ngx_event_module_t ngx_epoll_module_ctx = {& epoll_name, ngx_epoll_create_conf, ngx_epoll_init_conf, {ngx_epoll_add_event, ngx_epoll_del_event, ngx_epoll_add_event, ngx_epoll_del_event Ngx_epoll_add_connection, ngx_epoll_del_connection, # if (NGX_HAVE_EVENTFD) ngx_epoll_notify, # else NULL, # endif ngx_epoll_process_events, ngx_epoll_init, ngx_epoll_done,}} One of the init methods is ngx_epoll_init, where the epoll object is created and the ngx_event_actions method is set.

/ / file:src/event/modules/ngx_epoll_module.cstatic ngx_int_tngx_epoll_init (ngx_cycle_t * cycle, ngx_msec_t timer) {/ / create an epoll handle ep = epoll_create (cycle- > connection_n / 2); ngx_event_actions = ngx_epoll_module_ctx.actions } 2.2 Worker process initialization each module when the Worker process initializes, read the configuration information in ngx_worker_process_init to make some settings, and then call the init_process methods of all modules.

Let's take a look at the detailed code.

/ / file: src/os/unix/ngx_process_cycle.cstatic voidngx_worker_process_init (ngx_cycle_t * cycle, ngx_int_t worker) {. / / get configuration ccf = (ngx_core_conf_t *) ngx_get_conf (cycle- > conf_ctx, ngx_core_module) / / set priority setpriority (PRIO_PROCESS, 0, ccf- > priority) / / set file descriptor restrictions setrlimit (RLIMIT_NOFILE, & rlmt) setrlimit (RLIMIT_CORE, & rlmt) / / group and uid set initgroups (ccf- > username, ccf- > group) setuid (ccf- > user) / / CPU affinity cpu_affinity = ngx_get_cpu_affinity (worker) if (cpu_affinity) {ngx_setaffinity (cpu_affinity, cycle- > log);}. / call the init_process of each module to initialize the module for (I = 0; cycle- > modules [I]; iSuppli +) {if (cycle- > modules [I]-> init_process) {if (cycle- > modules [I]-> init_process (cycle) = = NGX_ERROR) {exit (2);}.} we mentioned ngx_event_core_module earlier, and its init_process method is ngx_event_process_init.

/ / file: src/event/ngx_event.cngx_module_t ngx_event_core_module = {ngx_event_process_init,}; in the ngx_event_process_init of ngx_event_core_module, we will see that the Worker process uses epoll_create to create epoll objects and epoll_ctl to listen for connection requests on listen socket.

Let's take a closer look at the code of ngx_event_process_init.

/ / file: src/event/ngx_event.cstatic ngx_int_t ngx_event_process_init (ngx_cycle_t * cycle) {/ / call the module's init to create the epoll object for (m = 0; cycle- > modules [m]; modules +) {if (cycle- > modules [m]-> type! = NGX_EVENT_MODULE) {continue;} module- > actions.init (cycle, ngx_timer_resolution) break } / / get the sokcet you are listening to, and add them all to the epoll ngx_event_t * rev ls = cycle- > listening.elts; for (I = 0; I

< cycle->

Listening.nelts; iTunes +) {/ / gets a ngx_connection_t c = ngx_get_connection (ls[ I] .fd, cycle- > log); / / sets the callback function to ngx_event_accept rev- > handler = ngx_event_accept if (ngx_add_event (rev, NGX_READ_EVENT, 0) = = NGX_ERROR) {return NGX_ERROR;} the handler of the READ event registered through ngx_add_event. Ngx_add_event is an abstraction, and for epoll it is just an encapsulation of epoll_ctl.

/ / file: src/event/ngx_event.h#define ngx_add_event ngx_event_actions.add//file: src/event/modules/ngx_epoll_module.cstatic ngx_int_t ngx_epoll_add_event () {if (epoll_ctl (ep, op, c-fd, & ee) =-1) {} TODO: epoll_create hasn't been solved yet.

2.3 entering epollwait in ngx_worker_process_init, both epoll_create and epoll_ctl have been completed. The next step is to enter the event loop and execute the epoll_wait to handle it.

/ / file: src/event/ngx_event.cvoidngx_process_events_and_timers (ngx_cycle_t * cycle) {/ / Anti-accept alarm group lock if (ngx_use_accept_mutex) {/ / attempts to acquire the lock. If you fail to acquire the lock, you will directly return if (ngx_trylock_accept_mutex (cycle) = = NGX_ERROR) {return; / / to acquire the lock successfully, then set the NGX_POST_EVENTS flag. If (ngx_accept_mutex_held) {flags | = NGX_POST_EVENTS; else {} / / handle various events (void) ngx_process_events (cycle, timer, flags);} at the beginning of the ngx_process_events_and_timers, determine whether to use the accpet_mutext lock. This is a solution to prevent surprises. If used, first call ngx_trylock_accept_mutex to acquire the lock. If the acquisition fails, it will be returned directly, and try again after a period of time. Sets the flag bit of NGX_POST_EVENTS if the success is obtained.

Next, call ngx_process_events to handle various network and timer events. For epoll, this function is an encapsulation of epoll_wait.

/ / file: src/event/ngx_event.h#define ngx_process_events ngx_event_actions.process_events//file: src/event/modules/ngx_epoll_module.cstatic ngx_int_t ngx_epoll_process_events (ngx_cycle_t * cycle,) {events = epoll_wait (ep, event_list, (int) nevents, timer); for I = 0; I events; iTunes + {if (flags & NGX_POST_EVENTS) {ngx_post_event (rev, queue) } else {/ / calls the callback function rev- > handler (rev);} it is visible that epoll_wait is called in ngx_epoll_process_events to wait for various events to occur. If there is no NGX_POST_EVENTS flag, call back rev- > handler directly for processing. If the accept_mutex lock is used, save the event first and go to accpet later when the time is right.

Briefly summarize the contents of this section. Only the bind and listen of socket are done in the Master process. A lot of things are done in the Worker process. Epoll is created and epoll_ctl is used to monitor the events of the socket in the state of listen. Finally, you call epoll_wait to enter the event loop and start dealing with various network and timer events. The process summary of this section is shown in the figure.

3. Here comes the user connection! Now assume that the user's connection request has arrived, and when epoll_wait returns, it will execute its corresponding handler function ngx_add_event.

When it is executed in the callback function, the socket representing the listen status is connected to. So this function does three main things.

1. Call accept to get the user connection

two。 Get the connection object, whose callback function is ngx_http_init_connection

3. Add the new connection socket to the epoll through epoll_ctl for management

Let's take a look at the ngx_event_accept detailed code.

/ / file: src/event/ngx_event_accept.cvoid ngx_event_accept (ngx_event_t * ev) {do {/ / receive the established connection s = accept (lc- > fd, & sa.sockaddr, & socklen); if s {/ / 3.1 get connection c = ngx_get_connection (s, ev- > log) / / 3.2 add a new connection if (ngx_add_conn (c) = = NGX_ERROR) {ngx_close_accepted_connection (c); return;} while (ev- > available); when a read event on listen socket occurs, it means that a user connection is ready. So you can take it out directly through accept. After taking out the connection, get a free connection object and add it to the epoll through ngx_add_conn for management.

To get connection, let's talk about ngx_get_connection. There is nothing to say about this function itself. Is to get a connection from the free_connections of ngx_cycle.

/ / file: src/core/ngx_connection.cngx_connection_t * ngx_get_connection (ngx_socket_t s, ngx_log_t * log) {c = ngx_cycle-free_connections; c-read = rev; c-write = wev; c-fd = s; c-log = log; return c} what is worth mentioning is the connection in free_connections. For HTTP service, it is initialized by ngx_http_init_connection. It sets the callback functions c-> read- > handler and c-> write- > handler for the read-write event of the connection.

/ / file: src/http/ngx_http_request.cvoid ngx_http_init_connection (ngx_connection_t * c) {rev = ngx_http_wait_request_handler; c-write-handler; rev-handler = ngx_http_wait_request_handler; c-write-handler = ngx_http_empty_handler;} 3.2.Add a new connection and let's take a look at ngx_add_conn, which is the function ngx_epoll_add_connection for epoll module.

/ / file: src/event/ngx_event.h#define ngx_add_conn ngx_event_actions.add_conn//file: src/event/modules/ngx_epoll_module.cstatic ngx_int_tngx_epoll_add_connection (ngx_connection_t * c) {struct epoll_event ee; ee.events = EPOLLIN | EPOLLOUT | EPOLLET | EPOLLRDHUP; ee.data.ptr = (void *) ((uintptr_t) c | c-read-instance) Epoll_ctl (ep, EPOLL_CTL_ADD, c-fd, & ee) c-read-active = 1; c-write-active = 1; return NGX_OK;} visible this is just an encapsulation of epoll_ctl. It is added here that if data arrives on the client connection socket, it will go to the ngx_http_wait_request_handler function registered in Section 3.1 above for processing. Then comes the processing logic of HTTP.

4. Summarize that there are not many network-related actions done in Nginx's Master, just create socket, and then bind and listen. Then it is to use their own fork to come out multiple Worker processes. Because every process is the same, each Worker has a socket handle to the listen state created by Master.

The Worker process handles a lot of network-related work. Epoll_create, epoll_ctl, and epoll_wait are all executed in the Worker process, including data read, processing, and write on the user connection.

1. First, create an epoll object using epoll_create

two。 Set callback to ngx_event_accept

3. Manage all socket events in listen state through epoll_ctl

4. Execute epoll_wait to wait for the connection on the listen socket to arrive

5. When a new connection arrives, epoll_wait returns and enters the ngx_event_accept callback.

In the 6.ngx_event_accept callback, new connections are also added to the epoll for management (the callback is ngx_http_init_connection)

7. Continue to enter epoll_wait waiting event

8. Enter the http callback function for processing when the user data request arrives

At this point, you can think that we have finished our discussion. Actually, there's one point we haven't taken into account. The process we discussed above is a case where Worker is working. Well, in the case of multi-Worker, we have not yet talked about the full picture of Nginx. From the above, we can see the following details:

1. Each Worker has its own epoll object

two。 Each Worker pays attention to all new connection events on the listen state

3. For a user connection, only one Worker will handle it, and the other Worker will not hold the socket of that user connection.

Based on these three conclusions, let's draw a full picture of Nginx.

Well, that's the end of today's sharing of Nginx network principles. I hope this excellent software can bring some inspiration and thinking to your work and help you improve your work. The students who can read here are all good. I'll add a drumstick to myself when I go back to dinner.

Finally, learn geek time and leave a thinking question.

Ponder question: "the epoll object in each Worker above listens to the same batch of socket in listen status, so when a user connection arrives, how can Nginx and Linux ensure that there are no surprise problems (only one Worker will respond to the request)? you are welcome to leave your thoughts in the comments section."

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.