Shulou(Shulou.com)06/02 Report--

This article mainly introduces Python how to achieve parent-son process sharing file object related knowledge, the content is detailed and easy to understand, the operation is simple and fast, has a certain reference value, I believe that after reading this Python how to achieve parent-son process sharing file object article will have a harvest, let's take a look.

Go directly to the code:

From multiprocessing import Process, Lock

Err_file = 'error1.log'

Err_fd = open (err_file,'w')

Def put (fd):

Print "PUT"

Fd.write ("hello, func put writen")

Print "END"

If _ _ name__=='__main__':

Paired list = []

For i in range (1):

P_list.append (Process (target=put, args= (err_fd,)

For p in p_list:

P.start ()

For p in p_list:

P.join ()

The intention of the above code is clear: derive a process through multiprocessing.Process to execute the put function, the function of the put function is also very clear, output PUT and END, and write "hello, func put write" to the file error1.log.

So in theory, the output should be the same as above, PUT and END, and then error1.log will have the phrase "hello, func put write". However, things are always a little unpredictable, and the result of code execution is:

[root@iZ23pynfq19Z ~] # py27 2.py; cat error1.log

PUT

END

[root@iZ23pynfq19Z ~] #

What!? Why is there nothing in error1.log?

Let's tweak the code a little bit and see what's amazing:

From multiprocessing import Process, Lock

Err_file = 'error1.log'

Err_fd = open (err_file,'w')

Def put (fd):

Print "PUT"

Fd.write ("hello, func put writen")

Fd.write ("o" * 4075) # Magic Line

Print "END"

If _ _ name__=='__main__':

Paired list = []

For i in range (1):

P_list.append (Process (target=put, args= (err_fd,)) for p in p_list:

P.start ()

For p in p_list:

P.join ()

Output result:

[root@iZ23pynfq19Z ~] # py27 2.py; cat error1.log

PUT

END

Hello, func put write

O.... (4075)

[root@iZ23pynfq19Z ~] #

Do you feel a sense of ignorance?

Today, two questions come to mind:

Why can't the first program write that sentence, but the second can?

What the hell is that 4075?

Before we explain these problems, we need to be clear about the characteristics of the standard IO library: full buffering, row buffering, and no buffering.

You can read the previous blog post: https://my.oschina.net/u/2291453/blog/806102

Because the file is being written now, the system IO will be fully buffered, that is, the buffer will be filled before it is brushed into the system write queue.

So the above problem is solved all at once, precisely because those mysterious' oases' fill the entire buffer, so the system brushes our content into the write queue, so 4075 how to come, is to use 4096-sizeof ("hello, func put writen") + 1, why + 1, because the buffer is not full, it has to be larger than to trigger write action.

So we can now get the answer, and if we want to write a file in a similar way in multiprcessing.Process, there are three ways to do it:

Full buffer

Manually call flush ()

Set the file object to be unbuffered

The first and second are already described above, so let's briefly talk about the third:

Taken from Python's official website Document:

Open (name [, mode [, buffering]])

...

The optional buffering argument specifies the file's desired buffer size: 0 means unbuffered

1 means line buffered, any other positive value means use a buffer of (approximately) that

Size (in bytes). A negative buffering means to use the system default, which is usually line

Buffered for tty devices and fully buffered for other files. If omitted, the system default is

Used. [2]

The figure above shows that if we are allowed to set buffering to 0 in open, then it is in unbuffered mode, so each write is written directly to the write queue instead of to the buffer. (the way with the lowest performance)

-I'm the cutting line-

After talking about the phenomenon and the ways to deal with it, we should go a little deeper.

I believe we have tried that the file can still be written normally when the closed file object is not displayed or when the call to flush is not displayed, so what is the matter?

In fact, when we close the program normally, the exit of the process will do some "tail-end" for us, such as closing open file descriptors, cleaning temporary files, cleaning memory, and so on. Because of this "good habit" of the system, our data can be brushed into the write queue when the file descriptor is turned off, and the file content will not be lost.

So based on this understanding, let's go back to the problem just now. When the child process calls put, theoretically, when the program exits, the closed file descriptor is not displayed, so the data is lost in the buffer.

Let's take a look at the realization of Process.

Multiprocessing/Processing.py

Def start (self):

''

Start child process

''

Assert self._popen is None, 'cannot start a process twice'

Assert self._parent_pid = = os.getpid ()

'can only start a process object created by current process'

Assert not _ current_process._daemonic

'daemonic processes are not allowed to have children'

_ cleanup ()

If self._Popen is not None:

Popen = self._Popen

Else:

From .forking import Popen

Self._popen = Popen (self)

_ current_process._children.add (self)

How does Popn do it again?

Multiprocessing/forking.py

Class Popen (object):

Def _ _ init__ (self, process_obj):

Sys.stdout.flush ()

Sys.stderr.flush ()

Self.returncode = None

Self.pid = os.fork ()

If self.pid = = 0:

If 'random' in sys.modules:

Import random

Random.seed ()

Code = process_obj._bootstrap ()

Sys.stdout.flush ()

Sys.stderr.flush ()

Os._exit (code)

The key point is the last os._exit (code). Why is it the most important? Because of the exit of this part, it will determine what the process will deal with.

What the heck is os._exit? In fact, it is the _ eixt of the standard library, so we can learn this thing easily again.

Https://my.oschina.net/u/2291453/blog/813259

In the link above, we can see more clearly that _ exit () and exit () are two different things. _ exit () is simple and violent, directly discarding user-mode content and entering the kernel, while exit () cleans up for us more patiently.

So can we assume: what would happen if the exit of Popen was not os._exit ()? Xiamen forklift Leasing Co., Ltd.

Luckily, sys.exit () is the exit () we want first. It's not too late. Just try it!

Multiprocessing/forking.py

Class Popen (object):

Def _ _ init__ (self, process_obj):

Sys.stdout.flush ()

Sys.stderr.flush ()

Self.returncode = None

Self.pid = os.fork ()

If self.pid = = 0:

If 'random' in sys.modules:

Import random

Random.seed ()

Code = process_obj._bootstrap ()

Sys.stdout.flush ()

Sys.stderr.flush ()

# os._exit (code)

Sys.exit (code)

Test the code and return to the original version without'o' padding

[root@iZ23pynfq19Z ~] # python 2.py; cat error1.log

PUT

END

Hello, func put write

We can see that it can indeed be written in, so as to prove that the above statement is tenable.

This is the end of the article on "how Python implements file objects shared by parent and child processes". Thank you for reading! I believe that everyone has a certain understanding of the knowledge of "how to achieve parent-son process sharing file objects in Python". If you want to learn more knowledge, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.