Reading a file vs loading a file into main memory from disk for processing
file, java
Solution
BufferReader buf = new BufferedReader(FileReader());
I presume that this is reading the file line by line from the disk. What is the advantage of this?
Not exactly. It is reading the file in chunks whose size is the default buffer size (8k bytes I think).
The advantage is that you don't need a huge heap to read a huge file. This is a significant issue since the maximum heap size can only be specified at JVM startup (with Hotspot Java).
You also don't consume the system's physical / virtual memory resources to represent the huge heap.
What is the advantage of loading the file directly into memory?
It reduces the number of system calls, and may read the file faster. How much faster depends on a number of factors. And you have the problem of dealing with really large files.
How do we do that in Java?
- Find out how large the file is.
- Allocate a byte (or character) array big enough.
- Use the relevant `read(byte[], int, int)` or `read(char[], int, int)` method to read the entire file.
You can also use a memory-mapped file ... but that requires using the `Buffer` APIs which can be a bit tricky to use.
I found some examples on Scanner or RandomAccessFile methods. Do they load the files into memory?
No, and no.
Should I use them? Which of the two should I use ?
Do they provide the functionality that you require? Do you need to read / parse text-based data? Do you need to do random access on a binary data?
Under normal circumstances, you should chose your I/O APIs based primarily on the functionality that you require, and secondarily on performance considerations. Using a `BufferedInputStream` or `BufferedReader` is usually enough to get acceptable* performance if you intend to parse it as you read it. (But if you actually need to hold the entire file in memory in its original form, then a `BufferedXxx` wrapper class actually makes reading a bit slower.)
* - Note that acceptable performance is not the same as optimal performance, but your client / project manager probably would not want your to waste time writing code to perform optimally ... if this is not a stated requirement.
Problem
how do I load a file into main memory? I read the files using, I use ``` BufferReader buf = new BufferedReader(FileReader()); ``` I presume that this is reading the file line by line from the disk. What is the advantage of this? What is the advantage of loading the file directly into memory? How do we do that in Java? I found some examples on `Scanner` or `RandomAccessFile` methods. Do they load the files into memory? Should I use them? Which of the two should I use ? Thanks in advance!!!