Shulou(Shulou.com)06/01 Report--

This article mainly introduces how python achieves multi-threaded concurrent crawling, which has certain reference value. Interested friends can refer to it. I hope you will gain a lot after reading this article. Let Xiaobian take you to understand it together.

Multithreaded concurrent fetching

Single thread is too slow, then you need to multi-thread, here to a simple thread pool template This program simply prints 1-10, but you can see that it is concurrent.

Although Python's multithreading is very useless, it can still improve efficiency to a certain extent for frequent web crawlers.

from threading import Threadfrom Queue import Queue time import sleep# q is the task queue #NUM is the total number of concurrent threads #JOBS is how many tasks q = Queue()NUM = 2JOBS = 10#Specific processing function, responsible for processing a single task def do_something_using(arguments): print arguments#This is a working process, responsible for constantly fetching data from the queue and processing def working():while True:arguments = q.get()do_something_using(arguments)sleep(1)q.task_done()#fork NUM threads waiting queue for i in range(NUM):t = Thread(target=working)t.setDaemon(True)t.start()#queue JOBS for i in range(JOBS):q.put(i)#wait for all JOBS to complete q.join() Thank you for reading this article carefully, I hope Xiaobian shared "Python how to achieve multi-threaded concurrent grab" This article is helpful to everyone, but also hope that you support, pay attention to industry information channels, more relevant knowledge waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.