Parsing 20 GB input file to an ArrayList
java
Solution
You shall use an external sorting algorithm, do not try to fit this in memory.
http://en.wikipedia.org/wiki/External_sorting
If you think it is too complex, try this:
- include H2 database in your project
- create a new on-disk database (will be created automatically on first connection)
- create some simple table where the numbers will be stored
- read data number-by-number and insert it into the database (do not forget to commit each 1000 numbers or so)
- select numbers with ORDER BY clause :)
- use JDBC resultSet to fetch results on-the-fly and write them to an output file
H2 database is simple, works very well with Java and can be embedded in your JAR (does not need any kind of installation or setup).
Problem
I need to sort a 20 GB file ( which consists of random numbers) in the ascending order, But I am not understanding what technique should I use. I tried to use ArrayList in my Java Program, but it runs out of Memory. Increasing the heap size didn't work too, I guess 20 GB is too big. Can anybody guide me, how should I proceed ?