Shulou(Shulou.com)06/02 Report--

This article mainly explains "Python process and thread knowledge point example analysis", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Next let the editor to take you to learn "Python process and thread knowledge point example analysis" bar!

Multithreading

A process contains at least one thread, but in fact, a process is made up of several threads. Threads are the execution units directly supported by the operating system, so high-level languages usually have built-in support for multithreading, and Python is no exception, and Python threads are real Posix Thread, not simulated threads.

Multithreading has the following advantages:

Threads can be used to put tasks in a program that have been occupied for a long time in the background.

The user interface can be more attractive, such as when the user clicks a button to trigger the handling of certain events, and a progress bar pops up to show the progress of the processing.

The program may run faster.

Threads are more useful in the implementation of some waiting tasks, such as user input, file reading and writing, and network data sending and receiving. In this case, we can release some precious resources such as memory footprint and so on.

Threads can be divided into:

Kernel threads: created and undone by the operating system kernel.

User thread: a thread implemented in a user program without kernel support.

Python's standard library provides two modules: _ thread, which is a low-level module, and threading, a high-level module that encapsulates _ thread. In most cases, you only need to use the threading module, and this module is also recommended.

Here, once again, take the download file as an example to implement it in a multithreaded way:

From random import randintfrom threading import Thread, current_threadfrom time import time, sleepdef download (filename): print ('thread% s is running...'% current_thread (). Name) print (' start downloading% s..% filename) time_to_download = randint (5,10) sleep (time_to_download) print ('% s download complete! % d seconds'% (filename, time_to_download) def download_multi_threading (): print ('thread% s is running...'% current_thread () .name) start = time () T1 = Thread (target=download, args= (' Python.pdf',), name='subthread-1') t1.start () T2 = Thread (target=download, args= ('nazha.mkv',) Name='subthread-2') t2.start () t1.join () t2.join () end = time () print ('spent% .3F seconds'% (end-start)) print ('thread% s is running...'% current_thread () .name) if _ name__ = =' _ main__': download_multi_threading ()

The way to implement multithreading is similar to that of multi-processes. The thread object is also created through the Thread class. The target parameter represents the function to be executed, the args parameter represents the parameter passed to the function, and then name names the current thread. The default names are Thread- 1, Thread-2, and so on.

In addition, any process starts a thread by default, which we call the main thread, and the main thread can start a new thread, and there is a function current_thread () in the threading module that returns an instance of the current thread. The name of the main thread instance is MainThread, and the name of the child thread is specified at the time of creation, that is, the name parameter.

Running result:

Thread MainThread is running...thread subthread-1 is running... Start downloading Python.pdf...thread subthread-2 is running... Start downloading nazha.mkv...nazha.mkv download complete! It took 5 seconds to complete the Python.pdf download! It took seven seconds, a total of 7.001 seconds, thread MainThread is running....

Lock

The biggest difference between multi-threads and multi-processes is that in multi-processes, there is a copy of the same variable in each process, which does not affect each other, while in multi-threads, all variables are shared by all threads, so, any variable can be modified by any thread, so the biggest danger of sharing data between threads is that multiple threads change a variable at the same time to change the content.

Here is an example of how multithreading can mess up memory by manipulating a variable at the same time:

From threading import Threadfrom time import time, sleep# assume this is your bank deposit: balance = 0def change_it (n): # deposit first and withdraw later The result should be 0: global balance balance = balance + n balance = balance-ndef run_thread (n): for i in range (100000): change_it (n) def nolock_multi_thread (): T1 = Thread (target=run_thread, args= (5,)) T2 = Thread (target=run_thread, args= (8,) t1.start t2.start () t1.join () t2.join () print (balance) if _ name__ ='_ main__': nolock_multi_thread ()

Running result:

-8

A shared variable balance is defined in the code, and then two threads are started, saving first and fetching later. In theory, the result should be 0. However, because the scheduling of threads is determined by the operating system, when T1 and T2 are executed alternately, as long as there are enough loops, the result of balance is not necessarily zero.

The reason is the following statement:

Balance = balance + n

The execution of this statement is divided into two steps:

First calculate balance + n and save the result to a temporary variable

Assign the value of a temporary variable to balance

That is, it can be seen as:

X = balance+nbalance=x

The normal operation is as follows:

Initial value balance = 0t1: X1 = balance + 5 # x1 = 0 + 5 = 5t1: balance = x1 # balance = 5t1: x1 = balance-5 # x1 = 5-5 = 0t1: balance = x1 # balance = 0t2: x2 = balance + 8 # x2 = 0 + 8 = 8t2: balance = x2 # balance = 8t2: x2 = balance-8 # x2 = 8-8 = 0t2: balance = x2 # balance = 0 result balance = 0

But in fact, the two threads run alternately, that is,

Initial value balance = 0t1: X1 = balance + 5 # x1 = 0 + 5 = 5t2: x2 = balance + 8 # x2 = 0 + 8 = 8t2: balance = x2 # balance = 8t1: balance = x1 # balance = 5t1: x1 = balance-5 # x1 = 5-5 = 0t1: balance = x1 # balance = 0t2: x2 = balance-8 # x2 = 0-8 = 8t2: balance = x2 # balance =-8 result balance =-8

To put it simply, it is because the modification to balance requires multiple statements, and when executing these statements, the thread may be interrupted, causing multiple threads to mess up the contents of the same object.

To ensure that the calculation is correct, you need to add a lock to change_it (), and after adding the lock, other threads must wait for the current thread to finish executing and release the lock before executing the function. And there is only one lock, no matter how many threads, at most one thread holds the lock at a time. It is realized by Lock of threading module.

So the code is modified to:

From threading import Thread, Lockfrom time import time, sleep# assume this is your bank deposit: balance = 0lock = Lock () def change_it (n): # deposit first and withdraw later The result should be 0: global balance balance = balance + n balance = balance-ndef run_thread_lock (n): for i in range (100000): # first acquire the lock: lock.acquire () try: # change it with ease: change_it (n) finally: # release the lock when you are done: lock.release () def nolock_multi_thread (): T1 = Thread (target=run_thread_lock, args= (5,)) T2 = Thread (target=run_thread_lock, args= (8) ) t1.start () t2.start () t1.join () t2.join () print (balance) if _ _ name__ ='_ _ main__': nolock_multi_thread ()

Unfortunately, Python does not fully play the role of multithreading, here you can write an endless loop, and then check the CPU utilization of the process through the task manager.

Normally, if you have two dead-loop threads, in a multicore CPU, you can monitor that it takes up 200% of the CPU, that is, two CPU cores.

If you want to run all the cores of N-core CPU, you must start N dead-loop threads.

The endless loop code is as follows:

Import threading, multiprocessingdef loop (): X = 0 while True: X = x ^ 1for i in range (multiprocessing.cpu_count ()): t = threading.Thread (target=loop) t.start ()

It can be monitored on the 4-core CPU that the CPU occupancy rate is only 102%, that is, only one core is used.

But if you rewrite the same dead loop with other programming languages, such as C, C++ or Java, you can directly fill all the cores, reaching 400% in 4 cores and 800% in 8 cores. Why not Python?

Because although the thread of Python is a real thread, when the interpreter executes the code, there is a GIL lock: Global Interpreter Lock, any Python thread must acquire the GIL lock before execution, and then, every 100bytecode executed, the interpreter automatically releases the GIL lock, giving other threads a chance to execute. This GIL global lock actually locks the execution code of all threads, so multi-threads can only be executed alternately in Python, even if 100 threads run on 100-core CPU, only one core can be used.

GIL is a historical legacy of Python interpreter design. Usually the interpreter we use is the official implementation of CPython. To really take advantage of multi-core, we must rewrite an interpreter without GIL.

Although multi-threading can not make full use of multi-core, it can greatly improve the running efficiency of the program. If you want to achieve multi-core tasks, you can achieve multi-core tasks through multi-processes. Multiple Python processes have their own independent GIL locks that do not affect each other.

ThreadLocal

When using multithreading, it is better for a thread to use its own local variables than global variables. As mentioned earlier, if there is no lock, multiple threads may change the value of a global variable. Local variables are visible only to each thread and will not affect other threads.

However, there is also a problem with the use of local variables, that is, when a function is called, it can be troublesome to pass, as shown below:

Def process_student (name): std = Student (name) # std is a local variable, but every function uses it, so it must be passed in: do_task_1 (std) do_task_2 (std) def do_task_1 (std): do_subtask_1 (std) do_subtask_2 (std) def do_task_2 (std): do_subtask_2 (std) do_subtask_2 (std)

Local variables need to be passed to each function layer by layer, which is troublesome. Is there a better way?

One idea is to use a global dict, and then use each thread as the key. The code example is as follows:

Global_dict = {} def std_thread (name): std = Student (name) # put std in the global variable global_dict: global_ input [threading. Current _ thread ()] = std do_task_1 () do_task_2 () def do_task_1 (): # do not pass std Instead, it is based on the current thread: std = global_ thread [threading.current _ thread ()]. Def do_task_2 (): # any function can find the std variable of the current thread: std = global_ thread [threading.current _ thread ()]

This approach is theoretically feasible, it can prevent local variables from passing in each layer of functions, but the code for getting local variables is not elegant enough. The local function is provided in the threading module, which can do this automatically. The code is as follows:

Import threading# creates a global ThreadLocal object: local_school = threading.local () def process_student (): # get the student associated with the current thread: std = local_school.student print ('Hello,% s (in% s)'% (std, threading.current_thread () .name) def process_thread (name): # bind ThreadLocal student: local_school.student = name process_student () T1 = threading.Thread (target= process_thread, args= ('Alice',)) Name='Thread-A') T2 = threading.Thread (target= process_thread, args= ('Bob',), name='Thread-B') t1.start () t2.start () t1.join () t2.join ()

Running result:

Hello, Alice (in Thread-A) Hello, Bob (in Thread-B)

A global variable local_school is defined in the code, which is a ThreadLocal object, and each thread can read and write student properties to it, but it does not affect each other and does not need to manage locks, which is handled internally by ThreadLocal.

ThreadLocal is most commonly used to bind a database connection, HTTP request, user identity information, etc., for each thread, so that all the processing functions called by a thread can easily access these resources.

Process vs thread

We have introduced the implementation of multi-process and multi-threading respectively, so which method should be chosen to implement concurrent programming, and what are the advantages and disadvantages of the two?

Usually in the implementation of multitasking, we design Master-Worker,Master to assign tasks and Worker to execute tasks, so in a multitasking environment, it is usually one Master and multiple Worker.

If you implement Master-Worker with multiple processes, the main process is Master and the other processes are Worker.

If you implement Master-Worker with multithreading, the main thread is Master and the other threads are Worker.

For multiple processes, the biggest advantage is high stability, because one child process is dead and will not affect the main process and other child processes. Of course, when the main process dies, all processes naturally hang up, but the main process is only responsible for assigning tasks, and the probability of hanging up is very low. The famous Apache first adopted the multi-process mode.

The disadvantages are:

It is expensive to create a process, especially in the windows system, while the Unix/ Linux system can call fork (), so the overhead is OK.

The number of processes that the operating system can run at the same time is limited, which is limited by memory and CPU.

For multithreading, it is usually too many processes, but not much faster; the disadvantage is poor stability, because all threads share the process's memory, and a thread hang-up can directly cause the entire process to crash. For example, on Windows, if there is something wrong with the code executed by a thread, you can often see the prompt: "the program has performed an illegal operation and is about to close." in fact, it is often a thread that has a problem, but the operating system will force the entire process to end.

Process / thread switching

The first thing to note about whether to adopt multitasking mode is that once there are too many tasks, the efficiency will not go up, mainly because it is costly to switch processes or threads.

The flow of the operating system when switching processes or threads is as follows:

First save the currently executed field environment (CPU register status, memory page, etc.)

Then prepare the execution environment for the new task (restore the last register state, switch memory pages, etc.)

Start the mission.

Although this switching process is very fast, it also takes time. If there are thousands of tasks, the operating system may be busy switching tasks and have no time to execute them. The most common situation is that the hard disk clicks loudly, the click window does not respond, and the system is suspended.

Compute intensive vsI/O intensive

The second consideration for multitasking is the type of task, which can be divided into compute-intensive and Imax-O-intensive.

The characteristic of computing-intensive task is that it takes a lot of computation and consumes CPU resources, such as encoding, decoding or format conversion of video, and so on. This kind of task depends on the computing power of CPU, although it can also be completed by multi-task, but the more tasks, the more time it takes to switch tasks, and the lower the efficiency of CPU task execution. Computing-intensive tasks mainly consume CPU resources, so it is usually inefficient to use a scripting language such as Python to execute such tasks. C language is the most suitable for this kind of task. We mentioned earlier that there is a mechanism for embedding Cmax Cure + code in Python. However, if you have to use Python to handle it, it is best to use multiple processes, and the number of tasks is preferably equal to the number of cores of CPU.

In addition to computing-intensive tasks, other tasks involving the network and storage media Icano can be regarded as Icano-intensive tasks, which are characterized by very low CPU consumption and spend most of their time waiting for the IUnip O operation to be completed (because the speed of Icando is much slower than that of CPU and memory). For I-CPU-intensive tasks, if you start multitasking, you can reduce the waiting time of I-map O and make it run efficiently. It is common to use multithreading to handle Imax O-intensive tasks.

Asynchronous IPUBO

The most important improvement of modern operating system is to support asynchronous Icano. If you make full use of the asynchronous Igamot O support provided by the operating system, you can use a single-process, single-threaded model to perform multitasking. This new model is called an event-driven model. Nginx is a Web server that supports asynchronous Iamp O. it can efficiently support multitasking by using a single-process model on a single-core CPU. On multicore CPU, you can run multiple processes (the same number as CPU cores) to take full advantage of multicore CPU. Server-side programs developed with Node.js also use this working mode, which is also a trend to achieve multitasking programming.

In Python, the programming model of single thread + asynchronous Icano is called co-programming. With the support of co-programming, we can write efficient multitasking programs based on event-driven. The biggest advantage of the cooperative program is the extremely high execution efficiency, because the subroutine switching is not thread switching, but is controlled by the program itself, so there is no thread switching overhead. The second advantage of the cooperative program is that there is no need for multithreaded locking mechanism, because there is only one thread, and there is no conflict between writing variables at the same time, and there is no need to lock the shared resources in the cooperative program, only to judge the state. Therefore, the execution efficiency is much higher than that of multithreading. If you want to make full use of the multi-core characteristics of CPU, the simplest method is multi-process + cooperative process, which not only makes full use of multi-core, but also gives full play to the high efficiency of cooperative program, and can achieve extremely high performance.

At this point, I believe that everyone on the "Python process and thread knowledge point example analysis" have a deeper understanding, might as well to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.