Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the method of convenient parallelism of Python code". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the method of convenient parallelism of Python code".

An example of tradition

A simple search for "Python multithreading tutorials" shows that almost all tutorials give examples of classes and queues:

Import os import PIL from multiprocessing importPool from PIL importImage SIZE = (75 folder 75) SAVE_DIRECTORY = 'thumbs' def get_image_paths (folder): return (os.path.join (folder, f) for f in os.listdir (folder) if'jpeg'in f) def create_thumbnail (filename): im = Image.open (filename) im.thumbnail (SIZE, Image.ANTIALIAS) base Fname = os.path.split (filename) save_path = os.path.join (base, SAVE_DIRECTORY, fname) im.save (save_path) if _ _ name__ = ='_ _ main__': folder = os.path.abspath ('11, 18, 2013, 2013, R000, IQM, Surface10d1958e7b766c3e840') os.mkdir (os.path.join (folder) SAVE_DIRECTORY)) images = get_image_paths (folder) pool = Pool () pool.map (creat_thumbnail, images) pool.close () pool.join ()

Ha, it looks a bit like Java, doesn't it?

I am not saying that it is wrong to use the producer / consumer model to deal with multithreaded / multiprocess tasks (in fact, this model has its own opportunities to use its talents). However, we can use a more efficient model when dealing with day-to-day scripting tasks.

The problem is...

First, you need a boilerplate class; second, you need a queue to pass objects; and, you also need to build methods on both sides of the channel to help it work (if you want to communicate in both directions or save the results, you need to introduce another queue).

The more worker, the more problems

Following this line of thinking, you now need a thread pool for worker threads. The following is an example from a classic IBM tutorial-speeding up web page retrieval through multi-threading.

# Example2.py''A more realistic thread pool example''import time import threading importQueue import urllib2 classConsumer (threading.Thread): def _ init__ (self, queue): threading.Thread.__init__ (self) self._queue = queue def run (self): whileTrue: content = self._queue.get () if isinstance (content) Str) and content = = 'quit': break response = urllib2.urlopen (content) print'Bye byescrafts' DefProducer (): urls = ['http://www.python.org',' http://www.yahoo.com' 'http://www.scala.org',' http://www.google.com' # etc.. ] Queue = Queue.Queue () worker_threads = build_worker_pool (queue, 4) start_time = time.time () # Add the urls to process for url in urls: queue.put (url) # Add the poison pillv for worker in worker_threads: queue.put ('quit') for worker in worker_threads: worker.join () print'Done! Time taken: {} '.format (time.time ()-start_time) def build_worker_pool (queue, size): workers = [] for _ in range (size): worker = Consumer (queue) worker.start () workers.append (worker) return workers if _ name__ = =' _ main__': Producer ()

This code works correctly, but take a closer look at what we need to do: construct different methods, trace a series of threads, and do a series of join operations to solve the annoying deadlock problem. This is just the beginning.

So far we have reviewed the classic multithreading tutorials, which are somewhat hollow, aren't they? Templated and error-prone, this style of getting half the effort with half the effort is obviously not so suitable for daily use, but fortunately we have a better way.

Why not try map

Map, a small and delicate function, is the key to the simple parallelization of Python programs. Map is derived from functional programming languages such as Lisp. It can realize the mapping between two functions through a sequence.

Urls = ['http://www.yahoo.com',' http://www.reddit.com'] results = map (urllib2.urlopen, urls)

The above two lines of code pass each element in the urls sequence as a parameter to the urlopen method and save all the results to the list of results. The results are roughly equivalent to:

Results = [] for url in urls: results.append (urllib2.urlopen (url))

The map function does a series of operations, such as sequence operation, parameter transfer and result saving.

Why is this important? This is because with the right libraries, map can easily parallelize operations.

There are two libraries in Python that contain map functions: multiprocessing and its little-known sublibrary multiprocessing.dummy.

Here's a few more words: multiprocessing.dummy? A threaded clone of the mltiprocessing library? Is this shrimp? Even in the official documentation of the multiprocessing library, there is only one description of this sublibrary. And this description is basically translated into adult saying: "well, there is such a thing, you know." Believe me, this library is grossly undervalued!

Dummy is a complete clone of the multiprocessing module, except that the multiprocessing acts on the process, while the dummy module acts on the thread (and therefore includes all the common multithreading limitations of Python). So it is extremely easy to replace these two libraries. You can choose different libraries for IO-intensive tasks and CPU-intensive tasks.

Give it a try

Use the following two lines of code to reference the library that contains parallelized map functions:

From multiprocessing importPool from multiprocessing.dummy importPoolasThreadPool

Instantiate the Pool object:

Pool = ThreadPool ()

This simple statement replaces the work of seven lines of code for the buildworkerpool function in example2.py. It generates a series of worker threads and initializes them, storing them in variables for easy access.

The Pool object has some parameters, and all I need to focus on here is its first parameter: processes. This parameter sets the number of threads in the thread pool. The default value is the number of cores of the current machine CPU.

In general, when performing CPU-intensive tasks, the more cores you call, the faster you will be. But when dealing with network-intensive tasks, things are a bit unpredictable, and it would be wise to experiment to determine the size of the thread pool.

Pool = ThreadPool (4) # Sets the pool size to 4

When there are too many threads, the time it takes to switch threads will even exceed the actual working time. For different jobs, it's a good idea to try to find the optimal thread pool size.

Once the Pool object is created, a parallelized program is about to emerge. Let's take a look at the rewritten example2.py

Import urllib2 from multiprocessing.dummy importPoolasThreadPool urls = ['http://www.python.org',' http://www.python.org/about/', 'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html',' http://www.python.org/doc/', 'http://www.python.org/download/', 'http://www.python.org/getit/', 'http://www.python.org/community/',' https://wiki.python.org/moin/', 'http://planet.python.org/',' https://wiki.python.org/moin/LocalUserGroups', 'http://www.python.org/psf/',' http://docs.python.org/devguide/', 'http://www.python.org/community/awards/' # etc.. ] # Make the Pool of workers pool = ThreadPool (4) # Open the urls in their own threads # and return the results results = pool.map (urllib2.urlopen, urls) # close the pool and wait for the work to finish pool.close () pool.join () there are only four lines of code that actually work, of which only one is critical. The map function easily replaces the previous example of more than 40 lines. To be more interesting, I counted the time consumed by different methods and different thread pool sizes. # results = [] # for url in urls: # result = urllib2.urlopen (url) # results.append (result) # #-VERSUS-#-4 Pool-# # pool = ThreadPool (4) # results = pool.map (urllib2.urlopen Urls) # #-8 Pool-# # pool = ThreadPool (8) # results = pool.map (urllib2.urlopen, urls) # #-13 Pool-# # pool = ThreadPool (13) # results = pool.map (urllib2.urlopen, urls)

Results:

# Single thread: 14.4 Seconds # 4 Pool: 3.1 Seconds # 8 Pool: 1.4 Seconds # 13 Pool: 1.3 Seconds

It's a great result, isn't it? This result also explains why it is necessary to determine the size of the thread pool through experiments. On my machine, the benefits are limited when the thread pool size is greater than 9.

Another real example.

Generate thumbnails of thousands of pictures

This is a CPU-intensive task and is well suited for parallelization.

Basic single-process version

Import os import PIL from multiprocessing importPool from PIL importImage SIZE = (75 folder 75) SAVE_DIRECTORY = 'thumbs' def get_image_paths (folder): return (os.path.join (folder, f) for f in os.listdir (folder) if'jpeg'in f) def create_thumbnail (filename): im = Image.open (filename) im.thumbnail (SIZE, Image.ANTIALIAS) base Fname = os.path.split (filename) save_path = os.path.join (base, SAVE_DIRECTORY, fname) im.save (save_path) if _ _ name__ = ='_ _ main__': folder = os.path.abspath ('11, 18, 2013, 2013, R000, IQM, Surface10d1958e7b766c3e840') os.mkdir (os.path.join (folder) SAVE_DIRECTORY)) images = get_image_paths (folder) for image in images: create_thumbnail (Image)

The main job of the above code is to traverse the picture files in the incoming folder, generate thumbnails one by one, and save these thumbnails to a specific folder.

On my machine, it takes 27.9 seconds to process 6000 pictures with this program.

If we use the map function instead of the for loop:

Import os import PIL from multiprocessing importPool from PIL importImage SIZE = 'thumbs' def get_image_paths (folder): return (os.path.join (folder, f) for f in os.listdir (folder) if'jpeg'in f) def create_thumbnail (filename): im = Image.open (filename) im.thumbnail (SIZE, Image.ANTIALIAS) base, fname = os.path.split (filename) save_path = os.path.join (base, SAVE_DIRECTORY) Fname) im.save (save_path) if _ _ name__ ='_ _ main__': folder = os.path.abspath ('11-18-18-13-13-13-12-13-13-13-13-13-13-13-13-13-13-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12-12) os.mkdir (os.path.join (folder, SAVE_DIRECTORY) images = get_image_paths (folder) pool = Pool () pool.map (creat_thumbnail, images) pool.close () pool.join ()

5.6 seconds!

Although only a few lines of code have been changed, we have significantly improved the execution speed of the program. In a production environment, we can select multi-process and multi-thread libraries for CPU-intensive tasks and IO-intensive tasks to further improve execution speed-this is also a good way to solve the deadlock problem. In addition, because the map function does not support manual thread management, it makes the related debug work extremely simple.

Thank you for reading, the above is the content of "what is the method of convenient parallelism of Python code". After the study of this article, I believe you have a deeper understanding of what the method of convenient parallelism of Python code is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.