C++ Performance writing and reading from disk

c++, performance

Solution

The first thing to do is to determine where the time is going. Formatting and parsing text isn't trivial, and can take some time, but so can the actual writing and reading, given the size of the file. The second thing is to determine how "portable" the data have to be: the fastest solution is almost certainly to `mmap` (or its Windows equivalent) the array to the file directly, and never read or write. This doesn't provide a portable representation, however, and even upgrading the compiler might make the data unreadable. (Unlikely for 32 bit integers today, but it has happened in the past).

In general, if the time is going to reading and writing, you will want to investigate using `mmap`. If it is going to formatting and parsing, you will want to investigate some sort of binary format—this could also help reading and writing if it makes the resulting files smaller. The simplest binary format, writing the values using the normal network standard, requires no more than:

void
writeInt( std::ostream& dest, int32_t integer )
{
    dest.put( (integer >> 24) & 0xFF );
    dest.put( (integer >> 16) & 0xFF );
    dest.put( (integer >>  8) & 0xFF );
    dest.put( (integer      ) & 0xFF );
}

int32_t
readInt( std::istream& source )
{
    int32_t results = 0;
    results  = source.get() << 24;
    results |= source.get() << 16;
    results |= source.get() <<  8;
    results |= source.get();
    return results;
}

(Some error checking obviously needs to be added.)

If many of the integers are actually small, you could try some variable length encoding, such as that used in Google Protocol Buffers. If most of your integers are in the range -64...63, this could result in a file only a quarter of the size (which again, will improve the time necessary to read and write).

Problem

Original source

Related problems

• How to write a large buffer into a binary file in C++, fast?
• How to write a large buffer into a binary file in C++, fast?