How to split big HashMap<String, Integer> into smaller ones
algorithm, data-structures, dictionary, java
Solution
Since you mentioned in a comment you want to store word frequences/occurancies, I suggest the following data structure:
Use a Tree. Each Node in the tree would hold a letter, and have a frequency value. Root would be the representation of the empty word, and each node would represent the word that is the path from the root. In this tree finding/updating a frequency takes as many steps as long the word is, independently from the number of words.
If this tree would be too big for memory, an easy partition would be the first level in the tree, which is the first letter of each word. You could store this in different files.
If you need finer granularity, you could use the first letter as a folder name for example, and the 2nd letter as the file name in those folders etc.
Problem
One `HashMap<String, Integer>` has size of 50,000 entry; moreover, it is not fit to the memory at once. I wish to split this big table into `int chunkSize = 1024` in size small table. As a result, I have tried to code a method but my naive approach is iterating over the big table and create a small one. However, naive iterative method is `O(n)` and it is open to bug because it is not using built-in Java methods, just iterating over the table. Do you have a opinion how to approach this problem so that solution is time-efficient and more depend on Java built in methods. UPDATE: I will use these smaller hashMap to feed to the Pipeline system. Pipeline system is designed with Pipeline Design Pattern. For each stage, some String operation and Text data mining algorithms will be applied. Splitting the Big HashMap will add value to the old application and future pipeline system. Actually, splitting operation is now mondetary for the pipeline system; however, for old application, I have started to read how to fine-tune HashMap internal structure.