Buffered files (for faster disk access)

Question

I am working with large files and writing directly to disk is slow. Because the file is large I cannot load it in a TMemoryStream. TFileStream is not buffered so I want to know if there is a custom library that can offer buffered streams or should I rely only on the buffering offered by OS. Is the OS buffering reliable? I mean if the cache is full an old file (mine) might be flushed from cache in order to make room for a new file. My file is in the GB range. It contains millions of records. Unfortunately, the records are not of fix size. So, I have to do millions of readings (between 4 and 500 bytes). The reading (and the writing) is sequential. I don't jump up and down into the file (which I think is ideal for buffering). In the end, I have to write such file back to disk (again millions of small writes). David provided the his personal library that provides buffered disk access. ``` Speed tests: Input file: 317MB.SFF Delphi stream: 9.84sec David's stream: 2.05sec ___________________…

Accepted Answer

For everybody's interest: Embarcadero added `TBufferedFileStream` (see the documentation) in the latest Release of Delphi 10.1 Berlin.

Unfortunately, I can't say how it competes with the solutions given here as I haven't bought the update yet. I am also aware of that the question was asked on Delphi 7 but I am sure the reference to Delphi's own implementation can be useful in the future.

Buffered files (for faster disk access)

Solution

Problem