Shulou(Shulou.com)06/01 Report--

Editor to share with you the Linux pipeline and famous pipeline example analysis, I believe that most people do not know much about it, so share this article for your reference, I hope you will learn a lot after reading this article, let's go to understand it!

Pipes and famous pipes

1. Overview of pipelines and related API applications

1.1 key concepts related to pipelines

Pipe is one of the original Unix IPC forms supported by Linux and has the following characteristics:

The pipeline is half-duplex and data can only flow in one direction; when communication is needed, two pipes need to be established.

Can only be used between parent-child processes or sibling processes (related processes)

A separate file system is formed: a pipe is a file for processes at both ends of the pipe, but it is not an ordinary file, it does not belong to a file system, but a separate file system that forms a separate file system and exists only in memory.

Reading and writing of data: what a process writes to the pipe is read by the process at the other end of the pipe. The written content is added to the end of the pipeline buffer each time, and the data is read out from the header of the buffer each time.

1.2 creation of pipes:

# include int pipe (int fd [2])

The two ends of the pipeline created by this function are in the middle of a process, which does not make much sense in practical application. Therefore, a process generally fork a child process after the pipeline is created by pipe (), and then realize the communication between the parent and child processes through the pipeline (therefore, it is not difficult to deduce that as long as there is a genetic relationship between the two processes, the kinship here refers to having a common ancestor. Can communicate in a pipeline way).

1.3 read and write rules for pipes:

The two ends of the pipe can be described by the descriptors fd [0] and fd [1], respectively. It is important to note that the two ends of the pipe are fixed to the task. That is, one end can only be used for reading, which is indicated by the description word fd [0], which is called the pipe read end; the other end can only be used for writing, represented by the description word fd [1], and called the pipe write end. An error occurs if you try to read data from the pipe writer or write data to the pipe reader. The general Imax O function of a file can be used for pipes, such as close, read, write, and so on.

Read data from the pipe:

If the write end of the pipe does not exist, the end of the data is considered to have been read, and the number of readout bytes returned by the read function is 0

When the write end of the pipeline exists, if the number of requested bytes is greater than PIPE_BUF, the number of existing data bytes in the pipeline is returned. If the number of requested bytes is not greater than PIPE_BUF, the number of existing data bytes in the pipeline is returned (in this case, the amount of data in the pipeline is less than the requested data amount), or the number of requested bytes is returned (in this case, the amount of data in the pipeline is not less than the requested data amount). Note: (PIPE_BUF is defined in include/linux/limits.h, and different kernel versions may vary. Posix.1 requires PIPE_BUF to be at least 512 bytes, compared with 4096 in red hat 7.2).

Verification of read rules for pipes:

/ * * readtest.c * * / # include # include

Write data to the pipe:

When writing data to the pipe, the linux will not guarantee the atomicity of the write, and the write process will attempt to write data to the pipe whenever there is a free area in the pipe buffer. If the read process does not read away the data in the pipeline buffer, the write operation will remain blocked.

Note: it makes sense to write data to the pipe only if the read end of the pipe exists. Otherwise, the process that writes data to the pipeline will receive a SIFPIPE signal from the kernel that the application can process or ignore (the default action is application termination).

Verification of the write rules of pipes 1: the dependence of the writer on the reader

# include # include main () {int pipe_fd [2]; pid_t

The output is: Broken pipe, because the reader of the pipe and all its fork () products has been closed. If you keep the reader in the parent process, that is, after writing the pipe, and then close the reader of the parent process, the pipe will be written normally. Readers can verify this conclusion by themselves. Therefore, there should be at least one process when writing data to the pipe, in which the pipe reader is not closed, otherwise the above error will occur (the pipe is broken, the process receives a SIGPIPE signal, and the default action is process termination)

Verification of write rules for pipes 2:linux does not guarantee atomicity verification of write pipes

# include # include # include main (int argc

Conclusion:

Writing is non-atomic when the number of writes is less than 4096!

If you change the number of bytes written to 5000 for both writes in the parent process, it is easy to draw the following conclusions:

When the amount of data written to the pipe is greater than 4096 bytes, the free space of the buffer is written (made up) until all the data is written, and if no process reads the data, it is blocked.

1.4 example of pipeline application:

Example 1: for shell

Pipes can be used for input-output redirection, which directs the output of one command directly to the input of another command. For example, when you type who │ wc-l in a shell program (Bourne shell, C shell, etc.), the corresponding shell program creates who and wc processes and pipes between the two processes. Consider the following command line:

The running result of $kill-l is shown in the attached one.

$kill-l | the running result of grep SIGRTMIN is as follows:

30) SIGPWR 31) SIGSYS 32) SIGRTMIN 33) SIGRTMIN+134) SIGRTMIN+2 35) SIGRTMIN

Example 2: for inter-process communication with kinship

The following example shows the specific application of the pipeline, in which the parent process sends some commands to the child process through the pipeline, and the child process parses the command and handles it accordingly.

# include # include main () {int pipe_fd [2]; pid_t

1.5 limitations of piping

The main limitations of the pipeline are reflected in its characteristics:

Only unidirectional data streams are supported

Can only be used between related processes

No name.

The buffer of the pipe is limited (the pipe system exists in memory and a page size is allocated to the buffer when the pipe is created)

What the pipeline transmits is an unformatted byte stream, which requires that the reader and writer of the pipeline must agree on the format of the data in advance, such as how many bytes are counted as a message (or command, or record), and so on.

2. Overview of famous pipelines and related API applications

2.1 key concepts related to well-known pipes

A major limitation of pipeline applications is that it has no name, so it can only be used for consanguineous inter-process communication, which was overcome after a named pipeline (named pipe or FIFO) was proposed. FIFO differs from pipes in that it provides a pathname associated with it and exists in the file system as a FIFO file. In this way, even processes that are not related to the creation process of the FIFO can communicate with each other through the FIFO as long as they can access the path (between the processes that can access the path and the creation process of the FIFO), so that data can be exchanged through processes that are not related to the FIFO. It is worth noting that FIFO strictly follows first-in, first-out (first in first out), and reads to pipes and FIFO always return data from the beginning, while writes to them add data to the end. They do not support file positioning operations such as lseek ().

2.2 creation of named pipes

# include # include int mkfifo (const char * pathname, mode_t mode)

The first argument to this function is a normal pathname, which is the name of the created FIFO. The second argument is the same as the mode parameter in the open () function that opens the normal file. If the first parameter of mkfifo is an existing pathname, an EEXIST error will be returned, so typical calling code will first check whether the error is returned, and if it does, just call the function that opens FIFO. The general file's Imax O function can be used for FIFO, such as close, read, write, and so on.

2.3 rules for opening a named pipe

The named pipe has one more open operation than the pipe: open.

The opening rule of FIFO:

If the current open operation opens the FIFO for read, if a corresponding process has already opened the FIFO for writing, the current open operation will return successfully; otherwise, it may block until the corresponding process opens the FIFO for writing (the blocking flag is set for the current open operation); or it returns successfully (no blocking flag is set for the current open operation).

If the current open operation opens the FIFO for write, and if there is already a corresponding process opening the FIFO for read, the current open operation will return successfully; otherwise, it may block until the corresponding process opens the FIFO for read (the blocking flag is set for the current open operation), or an ENXIO error is returned (no blocking flag is set for the current open operation).

For verification of the opening rule, see attachment 2.

2.4 rules for reading and writing of famous pipes

Read data from FIFO:

Convention: if a process blocks opening FIFO in order to read data from FIFO, the read operation within the process is said to be a read operation with the blocking flag set.

If a process writes and opens the FIFO and there is no data in the current FIFO, the read operation with the blocking flag set will remain blocked. For read operations without blocking flags set,-1 is returned, and the current errno value is EAGAIN, reminding you to try again later.

For read operations with blocking flags set, there are two reasons for blocking: there is data in the current FIFO, but other processes are reading it, and there is no data in the FIFO. The reason for unblocking is that there are new data writes in the FIFO, regardless of the amount of data written in the letter and the amount of data requested by the read operation.

The read open blocking flag only exerts an effect on the first read operation of the process. If there are multiple read operation sequences in this process, the other read operations to be performed will no longer block after the first read operation is awakened and the read operation is completed, even if there is no data in the FIFO when the read operation is performed (in this case, the read operation returns 0).

If no process writes to open FIFO, read operations with blocking flags set will block.

Note: if there is data in the FIFO, the read operation with the blocking flag set will not block because the number of bytes in the FIFO is less than the number of bytes requested. In this case, the read operation will return the existing amount of data in the FIFO.

Write data to FIFO:

Convention: if a process blocks opening FIFO in order to write data to FIFO, the write operation within the process is said to be a write operation with the blocking flag set.

For write operations with blocking flags set:

When the amount of data to be written is not greater than PIPE_BUF, linux will guarantee the atomicity of the write. If the pipe idle buffer is not enough to hold the number of bytes to be written at this time, go to sleep and do not start an one-time write operation until the buffer can hold the number of bytes to be written.

When the amount of data to be written is greater than PIPE_BUF, linux will no longer guarantee the atomicity of the write. As soon as there is a free area in the FIFO buffer, the write process attempts to write data to the pipe, and the write operation returns after writing all the requested data.

For writes that do not have a blocking flag set:

When the amount of data to be written is greater than PIPE_BUF, linux will no longer guarantee the atomicity of the write. After all FIFO free buffers are full, the write operation returns.

When the amount of data to be written is not greater than PIPE_BUF, linux will guarantee the atomicity of the write. If the current FIFO free buffer can hold the number of bytes requested to be written, it will be returned successfully after writing; if the current FIFO free buffer cannot hold the number of bytes requested to be written, an EAGAIN error will be returned, reminding you to write later.

Validation of FIFO read and write rules:

Here are two programs for reading and writing FIFO, and various FIFO read and write rules can be verified by properly adjusting few places in the program or the command line parameters of the program.

Program 1: a program for writing FIFO

# include # include

Program application description:

Compile the reader into two different versions:

Blocking read version: br

And the non-blocking read version of nbr

Compile the writer into two four versions:

Non-blocking and the number of bytes requested to write is greater than the PIPE_BUF version: nbwg

Non-blocking and the number of bytes requested to be written is not greater than PIPE_BUF version: version nbw

Blocking and the number of bytes requested to write is greater than the PIPE_BUF version: bwg

Blocking and the number of bytes requested to be written is not greater than PIPE_BUF version: version bw

Br, nbr and w will be used instead of blocking and non-blocking reads in the corresponding programs.

Verify blocking writes:

Non-atomicity when the amount of data requested to be written is greater than PIPE_BUF:

Nbr 1000

Bwg

Atomicity when the amount of data requested to be written is not greater than PIPE_BUF:

Nbr 1000

Bw

Verify non-blocking write operations:

Non-atomicity when the amount of data requested to be written is greater than PIPE_BUF:

Nbr 1000

Nbwg

Atomicity when the amount of data requested to be written is not greater than PIPE_BUF:

Nbr 1000

Nbw

Regardless of whether the write on blocking flag is set or not, the atomicity of the write is not guaranteed when the number of bytes requested to write is greater than 4096. But there is an essential difference between the two:

For blocking writes, the write operation waits after filling the free area of the FIFO until all the data is written, and the data requested to write will eventually be written to the FIFO

Non-blocking writes return (the actual number of bytes written) after filling up the free area of the FIFO, so some data cannot be written in the end.

The verification of read operations is relatively simple and will not be discussed.

2.5 well-known pipeline application example

After verifying the corresponding read and write rules, the application example seems unnecessary.

Summary:

Pipes are often used in two aspects: (1) pipes are often used in shell (redirection as input), in this application mode, the creation of pipes is transparent to users; and (2) for inter-process communication with kinship, users create pipes themselves and complete read and write operations.

FIFO can be said to be the promotion of the pipeline, overcoming the limitation of no name of the pipeline, so that the unrelated process can also use the first-in-first-out communication mechanism to communicate.

The data of pipeline and FIFO is a byte stream, and a specific transport "protocol" must be determined in advance between applications to propagate messages with specific meaning.

To apply pipelines and FIFO flexibly, understanding their reading and writing rules is the key.

The running result of 1:kill-l is attached, which shows all the signals supported by the current system:

1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL5) SIGTRAP 6) SIGABRT

In addition to being used here to illustrate pipeline applications, these signals will be classified and discussed in the following topics.

Attachment 2: verification of FIFO open rules (mainly verifying the dependency of write open on read open)

The above is all the contents of the article "example Analysis of pipes and famous pipes in Linux". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.