How do I read a large file gradually?
java, large-files
Solution
You need to open up a `Reader` (e.g. a `BufferedReader` wrapping an `InputStreamReader` wrapping a `FileInputStream`) and read chunks at a time with `read(char[], int, int)` or `read(char[])`. It's up to you to take care of finding the token - including in the case where it starts in one chunk and ends on another. Also be aware that `read()` may not fill the buffer; you need to use the return value to see how much data it's actually written to the array.
Problem
I'm having some problems reading a file with java. It is absolutely huge (2,5G) and adjusting my memory doesn't help. The data is all on a single line so I can't read it one line at a time. What I would like to do is to read the file until I find a certain string for example "<|start|>" or "<|end|>" and then print the data in between these strings so the memory is cleared and I can continue reading the rest of the file. So what I basically am looking for is a type of reader that starts reading at a certain start string and stops reading at a stop string. Can anyone help me?