Parsing Very Large XML Files Using Multiprocessing
multiprocessing, python, xml
Solution
For a file of that size, I suggest you use a streaming XML parser. In Python, this would be the `iterparse` method from cElementTree or lxml.etree:
http://effbot.org/zone/element-iterparse.htm
Problem
I have a huge XML file, and I'm a tad bit at a loss on how to handle it. It's 60 GBs, and I need to read it. I was thinking if there a way to use multiprocessing module to read the python file? Does anyone have any samples of doing this that they could point me to? Thank you