Shulou(Shulou.com)06/02 Report--

This article mainly explains the "introduction of three forms of python download file", the content of the explanation is simple and clear, easy to learn and understand, now please follow the editor's train of thought slowly in depth, together to study and learn the "introduction of the three forms of python download file"!

Depending on the situation, python downloads can be divided into three types:

Download small files

Large file download

Batch download

Python Mini File download

Process: use request.get to request a link, place the returned content in the variable r, and then write r to where you want it.

Python Mini File download process

Take downloading the above flowchart file as an example:

Python file download

After running, it will be saved to the new folder of the current folder.

Python large file download

When we download a small file, we temporarily store the contents of the file in a variable. Think about it. What's wrong with downloading a large file like this? Quite simply, if you have only 8 gigabytes of memory and the file you want to download is as big as 10 gigabytes, you won't be able to download it. And native software takes up a lot of memory. If you only have 8 gigabytes of memory, the remaining available memory may actually be lower than that of 2G-4G. How to download a large file in this case?

Streaming block download

Principle: write memory to a file piece by piece to avoid taking up too much memory.

Python large file download process

When request.get (stream=True) is set, it starts the stream mode download, which typically starts when the content of the r variable is called. The code is as follows:

Python large file download

Python batch file download

The so-called batch download, of course, is not a file download, for example, we want to download Baidu pictures, if a download will appear two negative situations:

If a request is blocked, the whole queue will be blocked.

If it is a small file, single-thread download is too slow

Our solution is to use an asynchronous strategy. If you can use the scrapy framework, it's a lot easier, because it combines the twisted asynchronous driver architecture, and you don't have to write asynchronous at all. However, our python practical treasure book is about tutorials, so let's tell you how to download asynchronously:

We need to use two packages, one is asyncio and the other is aiohttp. Asyncio is the original library of Python3, but aiohttp requires you to open it using cmd/Terminal and enter the following command to install:

Pip install aiohttp

Note that asyncio is a single-process concurrent library, not multi-threaded or multi-process. It simply runs asynchronously in a process. The switch is marked with await, and the function that can be switched is marked with async. For example, the code for downloading two images asynchronously is as follows:

Python asynchronous batch download

Note: img = await session.get (url)

At this time, when you request the first picture to get the data, it will switch to request the second picture or other picture, and then switch back after the first picture gets all the data. In order to achieve multi-threaded batch download function, the speed is super fast, download ultra-clear large pictures with this method can be one per second.

Thank you for your reading, the above is the content of the "introduction of the three forms of python download file". After the study of this article, I believe you have a deeper understanding of the introduction of the three forms of python download file, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.