Shulou(Shulou.com)06/01 Report--

Python multithreading and multiprocessing guidelines are like, I believe that many inexperienced people do not know what to do, so this article summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

Using Python to analyze data can sometimes greatly improve the speed of the program if you use the correct data structure and algorithm. One way to do this is to use Muiltithreading (multithreading) or Multiprocessing (multiprocessing).

Using Python to analyze data can sometimes greatly improve the speed of the program if you use the correct data structure and algorithm. One way to do this is to use Muiltithreading (multithreading) or Multiprocessing (multiprocessing).

Let's take an example and write a small Python script to download images from Unsplash. We will start by downloading the version of one image at a time. Next, we use threads to improve execution speed.

Multithreading and multiprocessing in Python

Multithreading

Simply put, threads allow you to run programs in parallel. Tasks that spend a lot of time waiting for external events are usually suitable for threading. They are also known as Bound O Bound tasks such as reading and writing from files, network operations, or online downloads using API. Let's look at an example that shows the benefits of using threads.

1. No threads

In this example, we want to run the program sequentially to see how long it takes to download 15 images from Unsplash API:

Import requests import time img_urls = ['https://images.unsplash.com/photo-1516117172878-fd2c41f4a759',' https://images.unsplash.com/photo-1532009324734-20a7a5813719', 'https://images.unsplash.com/photo-1524429656589-6633a470097c',' https://images.unsplash.com/photo-1530224264768-7ff8c1789d79', 'https://images.unsplash.com/photo-1564135624576-c5c88640f235', 'https://images.unsplash.com/photo-1541698444083-023c97d3f4b6', 'https://images.unsplash.com/photo-1522364723953-452d3431c267',' https://images.unsplash.com/photo-1513938709626-033611b8cc03', 'https://images.unsplash.com/photo-1507143550189-fed454f93097',' https://images.unsplash.com/photo-1493976040374-85c8e12f0c0e', 'https://images.unsplash.com/photo-1504198453319-5ce911bafcde', 'https://images.unsplash.com/photo-1530122037265-a5f1f91d3b99', 'https://images.unsplash.com/photo-1516972810927-80185027ca84',' https://images.unsplash.com/photo-1550439062-609e1531270e', 'https://images.unsplash.com/photo-1549692520-acc6669e2f0c'] start = time.perf_counter () # start timer for img_url in img_urls: img_name = img_url.split (' /') [3] # get image name from url img_bytes = requests.get (img_url) .content with open (img_name 'wb') as img_file: img_file.write (img_bytes) # save image to disk finish = time.perf_counter () # end timer print (f "Finished in {round (finish-start,2)} seconds") # results Finished in 23.101926751 seconds

It took 23 seconds.

two。 Multithreading

Let's take a look at how the threading module in Pyhton can significantly improve our program execution:

Import time from concurrent.futures import ThreadPoolExecutor def download_images (url): img_name = img_url.split ('/') [3] img_bytes = requests.get (img_url). Content with open (img_name Wb') as img_file: img_file.write (img_bytes) print (f "{img_name} was downloaded") start = time.perf_counter () # start timer with ThreadPoolExecutor () as executor: results = executor.map (download_images,img_urls) # this is Similar to map (func, * iterables) finish = time.perf_counter () # end timer print (f "Finished in {round (finish-start) 2)} seconds ") # results Finished in 5.544147536 seconds

We can see that using threaded code can be significantly faster than not using threaded code. From 23 to 5 seconds.

For this example, note that there is overhead when creating a thread, so it makes sense to use a thread for multiple API calls, not just a single call.

In addition, for intensive computing, such as data processing, image processing multiprocessing performs better than threads.

After reading the above, have you mastered the methods of multithreading and multiprocessing guidelines in Python? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.