Shulou(Shulou.com)06/01 Report--

Editor to share with you how to achieve multi-threaded crawler Python, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to understand it!

What is a thread?

Thread, also known as lightweight process, is the smallest unit that the operating system can schedule operations. It is included in the process and is the actual operating unit of the process. The thread itself does not own system resources, only a few resources that are essential to running, but it can share all the resources owned by the process with other threads that belong to the same process. One thread can create and undo another thread, and multiple threads in the same process can execute concurrently.

Why use multithreading?

Threads are independent and concurrent execution flows in the program. Threads in a process are less isolated than separated processes, sharing memory, file handles, and the state that other processes should have.

Because the division scale of threads is smaller than that of processes, the concurrency of multithreaded programs is high. The process has an independent memory unit in the process of execution, and multiple threads share memory, which greatly improves the running efficiency of the program.

Threads have higher performance than processes because threads in the same process have common characteristics that multiple threads share the virtual space of the same process. The environment shared by threads includes process code snippets, process public data, and so on. Using these shared data, it is easy to communicate between threads.

When the operating system creates a process, it must allocate independent memory space and a large number of related resources for the process, but it is much easier to create threads. Therefore, the performance of using multithreading to achieve concurrency is much higher than using multiple processes.

To sum up, using multithreaded programming has the following advantages:

Memory cannot be shared between processes, but it is easy to share memory between threads.

When the operating system creates a process, it needs to reassign system resources to the process, but the cost of creating a thread is much lower. Therefore, using multi-threads to achieve multi-task concurrent execution is more efficient than using multi-processes.

Python language has built-in multi-thread function support, rather than simply acting as a scheduling mode of the underlying operating system, thus simplifying the multi-thread programming of Python.

Advantages of multithreading

Multithreading is similar to executing several different programs at the same time, and multithreading has the following advantages:

Threads can be used to put tasks in a program that have been occupied for a long time in the background.

The user interface can be more attractive, such as when the user clicks a button to trigger the handling of certain events, and a progress bar pops up to show the progress of the processing.

The program may run faster.

Threads are more useful in the implementation of some waiting tasks, such as user input, file reading and writing, and network data sending and receiving. In this case, we can release some precious resources such as memory footprint and so on.

Each independent thread has an entrance to the program, a sequential execution sequence, and an exit to the program. However, threads cannot execute independently and must be stored in the application, which provides multiple thread execution control.

Each thread has its own set of CPU registers, called the thread's context, which reflects the status of the last time the thread ran the thread's CPU register.

The instruction pointer and the stack pointer register are the two most important registers in the thread context. The thread always runs in the context of the process. These addresses are used to indicate the memory in the process address space that owns the thread.

Threads can be preempted (interrupted).

Threads can be put on hold (also known as sleep) while other threads are running-this is the thread's concession.

Threads can be divided into:

Kernel threads: created and undone by the operating system kernel.

User thread: a thread implemented in a user program without kernel support.

Simple example:

Import timeimport threading # Import a multi-thread library def sing (): # create a singing method for i in range (1Power5): print ("singing") time.sleep (1) def dance (): # create a dancing method for i in range (1Power5): print ("dancing") time.sleep (1) def run (): T1 = threading.Thread (target=sing) # create thread 1 T2 = threading.Thread (target=dance) # create thread 2 t1.start () # running thread 1 t2.start () # running thread 2

Run () run screenshot

These are all the contents of the article "how to implement multithreaded crawlers in Python". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.