Python fastest way to read a large number of small files into memory?
python
Solution
For a similar problem I have used this simple piece of code:
import glob
for file in glob.iglob("*.html"):
with open(file) as f:
a = f.read()
iglob doesn't stores all file simultaneously, this is perfect with a huge directory. Remenber to close files after you have finished, the construct "with-open" make sure for you.
Problem
I'm trying to read a few thousands html files stored on disk. Is there any way to do better than; ``` for files in os.listdir('.'): if files.endswith('.html') : with (open) files as f: a=f.read() #do more stuffs ```