Shulou(Shulou.com)06/01 Report--

Today, I would like to share with you how to use a line of code to achieve Python parallel processing related knowledge, detailed content, clear logic, I believe that most people still know too much about this knowledge, so share this article for your reference, I hope you can learn something after reading this article, let's take a look at it.

An example of tradition

A simple search for "Python multithreading tutorials" shows that almost all tutorials give examples of classes and queues:

Import osimport PILfrom multiprocessing import Poolfrom PIL import ImageSIZE = 'thumbs'def get_image_paths (folder): return (os.path.join (folder, f) for f in os.listdir (folder) if' jpeg' in f) def create_thumbnail (filename): im = Image.open (filename) im.thumbnail (SIZE, Image.ANTIALIAS) base, fname = os.path.split (filename) save_path = os.path.join (base) SAVE_DIRECTORY, fname) im.save (save_path) if _ _ name__ ='_ _ main__': folder ='_ _ main__': folder = os.path.abspath ('11'18''2013' R000 QM 'Surgical Monopoly E10d1958e7b766c3e840') os.mkdir (os.path.join (folder, SAVE_DIRECTORY)) images = get_image_paths (folder) pool = Pool () pool.map (creat_thumbnail, images) pool.close () pool.join ()

Ha, it looks a bit like Java, doesn't it?

I am not saying that it is wrong to use the producer / consumer model to deal with multithreaded / multiprocess tasks (in fact, this model has its own opportunities to use its talents). However, we can use a more efficient model when dealing with day-to-day scripting tasks.

The problem is...

First, you need a sample class

Second, you need a queue to pass objects.

Also, you need to build methods on both sides of the channel to help it work (if you want to communicate in both directions or save the results, you need to introduce another queue).

(recommended tutorial: python tutorial)

The more worker, the more problems

Following this line of thinking, you now need a thread pool for worker threads. The following is an example from a classic IBM tutorial-speeding up web page retrieval through multi-threading.

# Example2.py'''A more realistic thread pool example'''import timeimport threadingimport Queueimport urllib2class Consumer (threading.Thread): def _ _ init__ (self, queue): threading.Thread.__init__ (self) self._queue = queue def run (self): while True: content = self._queue.get () if isinstance (content Str) and content = = 'quit': break response = urllib2.urlopen (content) print' Bye byesinherdef Producer (): urls = ['http://www.python.org',' http://www.yahoo.com' 'http://www.scala.org',' http://www.google.com' # etc.. ] Queue = Queue.Queue () worker_threads = build_worker_pool (queue, 4) start_time = time.time () # Add the urls to process for url in urls: queue.put (url) # Add the poison pillv for worker in worker_threads: queue.put ('quit') for worker in worker_threads: worker.join () print' Done! Time taken: {} '.format (time.time ()-start_time) def build_worker_pool (queue, size): workers = [] for _ in range (size): worker = Consumer (queue) worker.start () workers.append (worker) return workersif _ name__ = =' _ main__': Producer ()

This code works correctly, but take a closer look at what we need to do: construct different methods, trace a series of threads, and do a series of join operations to solve the annoying deadlock problem. This is just the beginning.

So far we have reviewed the classic multithreading tutorials, which are somewhat hollow, aren't they? Templated and error-prone, this style of getting half the effort with half the effort is obviously not so suitable for daily use, but fortunately we have a better way.

Why not try map

Map, a small and delicate function, is the key to the simple parallelization of Python programs. Map is derived from functional programming languages such as Lisp. It can realize the mapping between two functions through a sequence.

Urls = ['http://www.yahoo.com',' http://www.reddit.com'] results = map (urllib2.urlopen, urls)

The above two lines of code pass each element in the urls sequence as a parameter to the urlopen method and save all the results to the list of results. The results are roughly equivalent to:

Results = [] for url in urls: results.append (urllib2.urlopen (url))

The map function does a series of operations, such as sequence operation, parameter transfer and result saving.

Why is this important? This is because with the right libraries, map can easily parallelize operations.

There are two libraries in Python that contain map functions: multiprocessing and its little-known sublibrary multiprocessing.dummy.

Here's a few more words: multiprocessing.dummy? A threaded clone of the mltiprocessing library? Is this shrimp? Even in the official documentation of the multiprocessing library, there is only one description of this sublibrary. And this description is basically translated into adult saying: "well, there is such a thing, you know." Believe me, this library is grossly undervalued!

Dummy is a complete clone of the multiprocessing module, except that the multiprocessing acts on the process, while the dummy module acts on the thread (and therefore includes all the common multithreading limitations of Python). So it is extremely easy to replace these two libraries. You can choose different libraries for IO-intensive tasks and CPU-intensive tasks.

Give it a try

Use the following two lines of code to reference the library that contains parallelized map functions:

From multiprocessing import Poolfrom multiprocessing.dummy import Pool as ThreadPool

Instantiate the Pool object:

Pool = ThreadPool ()

This simple statement replaces the work of seven lines of code for the build*worker*pool function in example2.py. It generates a series of worker threads and initializes them, storing them in variables for easy access.

The Pool object has some parameters, and all I need to focus on here is its first parameter: processes. This parameter sets the number of threads in the thread pool. The default value is the number of cores of the current machine CPU.

In general, when performing CPU-intensive tasks, the more cores you call, the faster you will be. But when dealing with network-intensive tasks, things are a little unpredictable, and it would be wise to experiment to determine the size of the thread pool.

Pool = ThreadPool (4) # Sets the pool size to 4

When there are too many threads, the time it takes to switch threads will even exceed the actual working time. For different jobs, it's a good idea to try to find the optimal thread pool size.

Once the Pool object is created, a parallelized program is about to emerge. Let's take a look at the rewritten example2.py

Import urllib2from multiprocessing.dummy import Pool as ThreadPoolurls = ['http://www.python.org',' http://www.python.org/about/', 'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html',' http://www.python.org/doc/', 'http://www.python.org/download/',' http://www.python.org/getit/', 'http://www.python.org/community/', 'https://wiki.python.org/moin/',' http://planet.python.org/', 'https://wiki.python.org/moin/LocalUserGroups',' http://www.python.org/psf/', 'http://docs.python.org/devguide/',' http://www.python.org/community/awards/' # etc.. ] # Make the Pool of workerspool = ThreadPool (4) # Open the urls in their own threads# and return the resultsresults = pool.map (urllib2.urlopen, urls) # close the pool and wait for the work to finishpool.close () pool.join ()

There are only four lines of code that actually work, of which only one is critical. The map function easily replaces the previous example of more than 40 lines. To be more interesting, I counted the time consumed by different methods and different thread pool sizes.

# results = [] # for url in urls:# result = urllib2.urlopen (url) # results.append (result) # #-VERSUS-#-4 Pool-# # pool = ThreadPool (4) # results = pool.map (urllib2.urlopen, urls) # #-8 Pool-# # pool = ThreadPool (8) # results = pool.map (urllib2.urlopen Urls) # #-13 Pool-# # pool = ThreadPool (13) # results = pool.map (urllib2.urlopen, urls)

Results:

# Single thread: 14.4 Seconds# 4 Pool: 3.1 Seconds# 8 Pool: 1.4 Seconds# 13 Pool: 1.3 Seconds

It's a great result, isn't it? This result also explains why it is necessary to determine the size of the thread pool through experiments. On my machine, the benefits are limited when the thread pool size is greater than 9.

Another real example.

Generate thumbnails of thousands of pictures

This is a CPU-intensive task and is well suited for parallelization.

Basic single process version import osimport PILfrom multiprocessing import Poolfrom PIL import ImageSIZE = (75 filename 75) SAVE_DIRECTORY = 'thumbs'def get_image_paths (folder): return (os.path.join (folder, f) for f in os.listdir (folder) if' jpeg' in f) def create_thumbnail (filename): im = Image.open (filename) im.thumbnail (SIZE, Image.ANTIALIAS) base Fname = os.path.split (filename) save_path = os.path.join (base, SAVE_DIRECTORY, fname) im.save (save_path) if _ _ name__ = ='_ _ main__': folder = os.path.abspath ('11, 18, 2013, 2013, R000, IQM, Surface10d1958e7b766c3e840') os.mkdir (os.path.join (folder) SAVE_DIRECTORY)) images = get_image_paths (folder) for image in images: create_thumbnail (Image)

The main job of the above code is to traverse the picture files in the incoming folder, generate thumbnails one by one, and save these thumbnails to a specific folder.

On my machine, it takes 27.9 seconds to process 6000 pictures with this program.

If we use the map function instead of the for loop:

Import osimport PILfrom multiprocessing import Poolfrom PIL import ImageSIZE = 'thumbs'def get_image_paths (folder): return (os.path.join (folder, f) for f in os.listdir (folder) if' jpeg' in f) def create_thumbnail (filename): im = Image.open (filename) im.thumbnail (SIZE, Image.ANTIALIAS) base, fname = os.path.split (filename) save_path = os.path.join (base) SAVE_DIRECTORY, fname) im.save (save_path) if _ _ name__ ='_ _ main__': folder ='_ _ main__': folder = os.path.abspath ('11'18''2013' R000 QM 'Surgical Monopoly E10d1958e7b766c3e840') os.mkdir (os.path.join (folder, SAVE_DIRECTORY)) images = get_image_paths (folder) pool = Pool () pool.map (creat_thumbnail, images) pool.close () pool.join ()

5.6 seconds!

Although only a few lines of code have been changed, we have significantly improved the execution speed of the program. In a production environment, we can select multi-process and multi-thread libraries for CPU-intensive tasks and IO-intensive tasks to further improve execution speed-this is also a good way to solve the deadlock problem. In addition, because the map function does not support manual thread management, it makes the related debug work extremely simple.

These are all the contents of the article "how to implement Python parallel processing with one line of code". Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.