Shulou(Shulou.com)06/03 Report--

This article shows you how to use Socket accept from the Linux source code, the content is concise and easy to understand, absolutely can make your eyes bright, through the detailed introduction of this article, I hope you can get something.

Look at the accept preface of Socket (TCP) from the source code of Linux

The author has always felt that it is an Exciting thing to know every piece of code from the application to the framework and then to the operating system. Today, from the perspective of Linux source code, the author looks at what Socket on the server side has done during Accept (based on the Linux 3.10 kernel).

The simplest example on the Server side

As we all know, the establishment of a server-side Socket requires four steps: socket, bind, listen and accept. Today, the author focuses on accept.

The code is as follows:

Void start_server () {/ / server fd int sockfd_server; / / accept fd int sockfd; int call_err; struct sockaddr_in sock_addr;. Call_err=bind (sockfd_server, (struct sockaddr*) (& sock_addr), sizeof (sock_addr)); Call_err=listen (sockfd_server,MAX_BACK_LOG); While (1) {struct sockaddr_in* s_addr_client = mem_alloc (sizeof (struct sockaddr_in)); int client_length = sizeof (* s_addr_client); / / this is our focus today accept sockfd = accept (sockfd_server, (struct sockaddr_ *) (s_addr_client), (socklen_t *) & (client_length)) If (sockfd = =-1) {printf ("Accept error!\ n"); continue;} process_connection (sockfd, (struct sockaddr_in*) (& s_addr_client));}}

First we create a Socket through the socket system call, where SOCK_STREAM is specified, and the last parameter is 0, which creates a normal all-TCP Socket. Here, we directly give the ops corresponding to TCP Socket, that is, the operation function.

Accept system call

All right, let's go directly to the accept system call.

# include / / success, return the descriptor representing the new connection, error returns-1, and the error code is set to errnoint accept (int sockfd,struct sockaddr* addr,socklen_t * addrlen); / / Note that Linux actually has an additional flags parameter added by the accept extension accept4:// to set O_NONBLOCK for the new connection descriptor | O_CLOEXEC (close after exec) these two tags int accept4 (int sockfd,struct sockaddr* addr,socklen_t * addrlen, int flags)

Note that the accept call here is wrapped by glibc in SYSCALL_CANCEL, which corrects the return value to only 0 and-1, and sets the absolute value of the error code in the errno. Since glibc's encapsulation of system calls is too complex, I won't go into details here. If you are looking for specific logic, use the

/ / pay attention to the space between accept and (, otherwise you will not find accept (int).

Just search the entire glibc code.

The key to understanding accept is that it creates a new Socket that connects to the peer Socket running connect () on the peer, as shown in the following figure:

Next, let's go to the Linux kernel source stack.

Accept |-> SYSCALL_CANCEL (accept.). | |-> SYSCALL_DEFINE3 (accept / / finally calls sys_accept4 |-> sys_accept4 / * checks whether the listening descriptor FD exists, but does not exist. | Return-BADF |-> sockfd_lookup_light |-> sock_alloc / * New Socket*/ |-> get_unused_fd_flags / * get an unused fd*/ |-> sock- > ops- > accept (sock...) / * calling core * /

The above process is shown below:

From this, we know that the core function is on sock- > ops- > accept, and since we focus on TCP, its implementation is inet_stream_ops- > accept, that is, inet_accept, which tracks the call stack again:

Sock- > ops- > accept |-> inet_steam_ops- > accept (inet_accept) / * you can see sk_prot=tcp_prot from the initial sock diagram |-> sk1- > sk_prot- > accept |-> inet_csk_accept

Well, after going through layers of packaging, we finally come to the specific logic part. The above code:

Struct sock * inet_csk_accept (struct sock * sk, int flags, int * err) {struct inet_connection_sock * icsk = inet_csk (sk); / * get the accept queue currently listening to sock * / struct request_sock_queue * queue = & icsk- > icsk_accept_queue;. / * if the listening Socket status is not TCP_LISEN, error * / if (sk- > sk_state! = TCP_LISTEN) goto out_err / * if the current accept queue is empty * / if (reqsk_queue_empty (queue)) {long timeo = sock_rcvtimeo (sk, flags & O_NONBLOCK) / * if it is non-blocking mode, return-EAGAIN * / error =-EAGAIN; if (! timeo) goto out_err directly / * if it is in blocking mode and the switch timeout is not 0, wait for the new connection to enter the queue * / error = inet_csk_wait_for_connect (sk, timeo); if (error) goto out_err } / * accept queue is not empty here, get a connection from queue * / req = reqsk_queue_remove (queue); newsk = req- > sk; / * fastopen judgment logic * /. / * return the new sock, that is, the sock derived from accept and equivalent to the client * / return newsk}

The above process is shown in the following figure:

Let's take a look at the timeout logic of inet_csk_wait_for_connect, or accept:

Static int inet_csk_wait_for_connect (struct sock * sk, long timeo) {for (;;) {/ * does not have a shock effect when calling accept in BIO by adding the EXCLUSIVE flag * / prepare_to_wait_exclusive (sk_sleep (sk), & wait, TASK_INTERRUPTIBLE) If (& icsk- > icsk_accept_queue) timeo = schedule_timeout (timeo);. Err =-EAGAIN; / * this timeout returns-EAGAIN * / if (! timeo) break;} finish_wait (sk_sleep (sk), & wait); return err;}

With the exclusice flag, we won't be surprised when we call accept in BIO (without epoll/select, etc.). You know from the code that it is EAGAIN rather than ETIMEOUT that returns (errno) when accept times out.

EPOLL (in accept) "surprise group"

Because in EPOLL LT (horizontal trigger mode), an accept event may wake up multiple (epoll_wait) threads waiting on this listen fd, and in the end, only one may successfully obtain a new connection (newfd), and the others are-EGAIN, that is, some unnecessary threads are awakened and done useless work. About the principle of epoll, you can take a look at the author's previous blog "looking at epoll from the linux source code":

Https://my.oschina.net/alchemystar/blog/3008840

To describe the reason here, the core is that when triggered horizontally, the epoll_wait will plug back into the fd while there are still unhandled events in the ready_list and wake up another process waiting on the epoll!

So we can see that although adding exclusive to yourself during epoll_wait will not alarm the group when an interrupt event is triggered, the horizontal trigger mechanism does cause a similar "surprise group" phenomenon!

From the above discussion, we can see that there are still events in fd1 that are the cause of the extra awakening, which is easy to understand. after all, this event is handled by another thread, which reckons that it is too late to run it.

Let's take a look at how to determine that this fd (the fd of listen sock) still has an unhandled event in the accept event.

/ / determine epi- > ffd.file- > fumbopp-> poll by fadopop.Following poll |-> tcp_poll / * if sock is in listen status Then the following function is responsible for * / |-> inet_csk_listen_poll / * to determine whether there are any unhandled events in the listening sock by whether the accept_queue queue is empty * / static inline unsigned int inet_csk_listen_poll (const struct sock * sk) {return! reqsk_queue_empty (& inet_csk (sk)-> icsk_accept_queue)? (POLLIN | POLLRDNORM): 0;}

Then we can draw the sequence diagram according to the logic.

In fact, not only accept, if multi-threaded epoll_wait read/write of the same fd is the same shock group, but no one should do so.

It is precisely because of this "shock group" effect that we often use a single thread to specialize in the form of accept, such as reactor mode. However, if a large number of connections pour in in an instant, there is still a bottleneck in single-threading processing, which can not make full use of the advantages of multi-cores, so it seems a little powerless in the massive short connection scenario. There is also a solution!

Using so_reuseport to solve the shock group

As mentioned earlier, we have this problem because we run epoll_wait on the same fd with multiple threads, so we can solve the problem by opening a few more fd. The first thing that comes to mind is to open a few more port numbers and artificially listen to the fd separately, but this obviously brings additional complexity. To solve this problem, Linux provides the parameter so_reuseport, which works as shown in the following figure:

Multiple fd listens to the same port number, does load balancing (Sharding) in the kernel, and distributes the tasks of accept to different Socket of different threads (Sharding). There is no doubt that the multi-core capability can be used to greatly improve the Socket distribution ability after a successful connection. Then our threading model can also be changed to multithreaded accept, as shown in the following figure:

Accept_queue fully connected queue

In the previous discussion, accept_queue is a core member of the accept system call, so how is the accept_queue add? The following figure shows the transition of accept_queue (full join queue) and syn_table semi-join hash table between client and server during three interactions. After the accept_queue is populated, the user thread gets the corresponding fd from the queue through the accept system call. It is worth noting that when the user thread is too late to process, the kernel will drop the successful connection of the three-way handshake, resulting in some strange phenomena.

Https://my.oschina.net/alchemystar/blog/3098219

In addition, for the specific filling mechanism and source code of accept_queue, you can see the detailed analysis of another blog, "looking at the listen and connection queue of Socket (TCP) from Linux source code":

Https://my.oschina.net/alchemystar/blog/4672630 the above content is how to use the accept of Socket from the Linux source code. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.