How to use large dicts in Python which not fit in memory?
data-structures, dictionary, memory, python
Solution
I couldn't find any (fast) module to do this and decided to create my own (my first python project - thanks @blubber for some ideas :P). You can find it on GitHub: https://github.com/ddofborg/diskdict Comments are welcome!
Problem
We use a `dict` which contains about 4GB of data for data processing. It's convenient and fast. The problem we are having is that this `dict` might grow over 32GB. I'm looking for a way to use a `dict` (just like a variable with get()-method etc) which can be bigger than the available memory. It would be great if this `dict` somehow stored the data on disk and retrieved the data from disk when `get(key)` is called and value for the `key` is not in memory. Preferably I wouldn't like to use an external service, like a SQL database. I did find Shelve, but it seems to need the memory too. Any ideas on how to approach this problem?