Shulou(Shulou.com)06/01 Report--

This article mainly shows you "python how to deal with large json files", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "python how to deal with large json files" this article.

As shown in the figure, the data to be processed is an json array, and it is very large

The following picture shows the configuration of the computer. Using the json.load () method to load the above json file, the computer is directly jammed.

The solution is as follows:

First preprocess with python, that is, a whole json data is loaded slowly, then load separately, one json object at a time, and then use the json.dumps () method to store it in txt for easy reading. Of course, it can also be stored in memory, depending on the situation.

Algorithm idea: because json is a kind of structured data, {} exists in pairs. Because the json file is read line by line, it is determined whether a complete json object has been read based on the number of {}, and if it is complete, it is stored in txt using the json.dumps () method.

GetUsefullData (temp + line,id) this method is to extract the data you need.

Def jsonProcess (fileName,round): id = 1 with open (fileName, 'ritual, encoding='utf-8') as f open ('.. / tempData/tempFile'+str (round) + '.txt') 'w') as tempFile: line = f.readline () line = f.readline () temp =' 'khNum = 0 id = 1 while line: # print (line) if line.find (' {')! =-1: khNum + = 1 if line.find ('}')! =-1 KhNum-= 1 if khNum = = 0: line = line.replace (' ',') data = getUsefullData (temp + line Id) id + = 1 if len (data) > 1: # print (data) try: tempFile.write (json.dumps (data) +'\ n') except 'json.decoder.JSONDecodeError': break Temp =''line = f.readline () continue temp + = line line = f.readline ()

Read method:

With open ('.. / tempData/tempFile'+str (round) + '.txt') as f: dataList = f.readlines () # parse the saved list for data in dataList: data = json.loads (data) these are all the contents of the article "how python handles large json files". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.