How to read line-delimited JSON from large file (line by line)
json, large-files, parsing, python
Solution
Just read each line and construct a json object at this time:
with open(file_path) as f:
for line in f:
j_content = json.loads(line)
This way, you load proper complete json object (provided there is no `\n` in a json value somewhere or in the middle of your json object) and you avoid memory issue as each object is created when needed.
There is also this answer.:
https://stackoverflow.com/a/7795029/671543
Problem
I'm trying to load a large file (2GB in size) filled with JSON strings, delimited by newlines. Ex: ``` { "key11": value11, "key12": value12, } { "key21": value21, "key22": value22, } … ``` The way I'm importing it now is: ``` content = open(file_path, "r").read() j_content = json.loads("[" + content.replace("}\n{", "},\n{") + "]") ``` Which seems like a hack (adding commas between each JSON string and also a beginning and ending square bracket to make it a proper list). Is there a better way to specify the JSON delimiter (newline `\n` instead of comma `,`)? Also, `Python` can't seem to properly allocate memory for an object built from 2GB of data, is there a way to construct each `JSON` object as I'm reading the file line by line? Thanks!