How to realize Multi-thread concurrent fetching by python

This article mainly shows you "python how to achieve multithreaded concurrent fetching", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "python how to achieve multithreaded concurrent crawling" this article.

Multithreaded concurrent fetching

If a single thread is too slow, you will need multithreading. Here is a simple thread pool template. This program simply prints 1-10, but you can see that it is concurrent.

Although the multithreading of Python is very creepy, it can improve the efficiency to a certain extent for the frequent network type of crawlers.

From threading import Threadfrom Queue import Queuefrom time import sleep# Q is the task queue # NUM is the total number of concurrent threads # JOBS is the number of tasks Q = Queue () NUM = 2JOBS = 1 task specific processing function, responsible for handling a single task def do_somthing_using (arguments): print arguments# this is the worker process Responsible for continuously fetching data from the queue and processing def working (): while True: arguments = q.get () do_somthing_using (arguments) sleep (1) q.task_done () # fork NUM thread waiting queue for i in range (NUM): t = Thread (target=working) t.setDaemon (True) t.start () # queue for i in range (JOBS) : q.put (I) # waiting for all JOBS to complete q.join () these are all the contents of the article "how to achieve multithreaded concurrent fetching by python" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!