How to read large files by Pandas
This article mainly introduces Pandas how to read large files, the article is very detailed, has a certain reference value, interested friends must read it!
How to use pandas to read large files, four tips:
How to use read_csv function to read files without headers
Get_chunk () method to read data in blocks
The concat () method overlays the database (vertically)
If the amount of data is too large, take random sampling (whether or not to put it back)
Filepath = open ("taobao.csv", errors= "ignore") # specify the file path reader = pd.read_csv (filepath, header=None, names= ["user_id", "item_id", "cat_id", "type", "time"], # specify column attribute name iterator=True) # loop,chunkSize,chunks = True, 10000000 [] # continuous assignment statement loop = TruechunkSize = 10000000chunks = [] while loop: # loop has always been True Execute loop try: chunk = reader.get_chunk (chunkSize) chunks.append (chunk) except StopIteration: loop = False print ("Iteration is stopped.") # if you consider that the amount of data is too large, only part of the data is extracted for analysis Take the method of not putting back the sampling # pd.concat (chunks, ignore_index=True). Sample (frac=0.05, replace=False) do not put back the sample, record not repeat df = pd.concat (chunks, ignore_index=True) these are all the contents of the article "how Pandas reads large files". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!
