Shulou(Shulou.com)06/01 Report--

This article will explain in detail how python streams to read G super-large files. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

How to read G super-large files by stream

Use with...open... You can read data from a file, which is familiar to all Python developers.

But if you use it improperly, it will also cause a lot of trouble.

For example, when you use the read function, in fact, Python will load all the contents of the file into memory at once. If the file has 10 gigabytes or more, then your computer will consume a lot of memory.

# read with open ("big_file.txt", "r") as fp: content = fp.read ()

For this problem, you may want to use readline to make a generator to return row by line.

Def read_from_file (filename): with open (filename, "r") as fp: yield fp.readline ()

But if the content of this file is only one line, one line is only 10 gigabytes, in fact, you will still read all the contents at once.

The most elegant solution is to specify that only a fixed size of content is read at a time when using the read method, as in the following code, only 8kb returns are read at a time.

Def read_from_file (filename, block_size = 1024 * 8): with open (filename, "r") as fp: while True: chunk = fp.read (block_size) if not chunk: break yield chunk

There is no problem with the function of the above code, but the code still looks a little bloated.

The code can be optimized with the help of partial functions and iter functions.

From functools import partial def read_from_file (filename, block_size = 1024 * 8): with open (filename, "r") as fp: for chunk in iter (partial (fp.read, block_size), "): yield chunk's article on" how to stream read G large files by python "ends here. I hope the above content can be of some help to you, so that you can learn more knowledge. If you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.