Shulou(Shulou.com)06/01 Report--

This article will explain in detail how python streams to read large files. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

How to read large files by stream

Use with... Open... You can read data from a file, which is familiar to all Python developers.

However, if it is not used properly, it will also cause a lot of trouble.

For example, when you use the read feature, Python loads the contents of the file into memory all at once. If the file has 10 GB or more, the computer will consume a very large amount of memory.

For this problem, you might consider using readline as a generator to return row by line.

However, if the contents of this file are on one line, you will still read everything at once for 10 GB per line.

The most elegant solution is to use the read method to specify that only a fixed size of content is read at a time. For example, in the following code, only 8kb is returned at a time.

The above code is functionally fine, but it still looks a little swollen.

Using partial and iterative functions, you can optimize your code like this

This is the end of this article on "how to stream python to read large files". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.