How to realize Multi-thread concurrent fetching by python

This article mainly introduces how python achieves multi-threaded concurrent crawling, which has certain reference value. Interested friends can refer to it. I hope you will gain a lot after reading this article. Let Xiaobian take you to understand it together.

Multithreaded concurrent fetching

Single thread is too slow, then you need to multi-thread, here to a simple thread pool template This program simply prints 1-10, but you can see that it is concurrent.

Although Python's multithreading is very useless, it can still improve efficiency to a certain extent for frequent web crawlers.

from threading import Threadfrom Queue import Queue time import sleep# q is the task queue #NUM is the total number of concurrent threads #JOBS is how many tasks q = Queue()NUM = 2JOBS = 10#Specific processing function, responsible for processing a single task def do_something_using(arguments): print arguments#This is a working process, responsible for constantly fetching data from the queue and processing def working():while True:arguments = q.get()do_something_using(arguments)sleep(1)q.task_done()#fork NUM threads waiting queue for i in range(NUM):t = Thread(target=working)t.setDaemon(True)t.start()#queue JOBS for i in range(JOBS):q.put(i)#wait for all JOBS to complete q.join() Thank you for reading this article carefully, I hope Xiaobian shared "Python how to achieve multi-threaded concurrent grab" This article is helpful to everyone, but also hope that you support, pay attention to industry information channels, more relevant knowledge waiting for you to learn!