reading and writing in chunks on linux using c

c, linux

Solution

Yes, `fread` will fetch the entire thing at once. (Assuming it's a regular file.) But it won't read 105 MB unless the file itself is 105 MB, and if you don't check the return value you have no way of knowing how much data was actually read, or if there was an error.

Use `fgets` (see `man fgets`) instead of `fread`. This will search for the line breaks for you.

char linebuf[1000];
FILE *file = ...;
while (fgets(linebuf, sizeof(linebuf), file) {
    // decode one line
}

There is a problem with your code.

char buffer[104857600]; // too big

If you try to allocate a large buffer (105 MB is certainly large) on the stack, then it will fail and your program will crash. If you need a buffer that big, you will have to allocate it on the heap with `malloc` or similar. I'd certainly keep stack usage for a single function in the tens of KB at most, although you could probably get away with a few MB on most stock Linux systems.

As an alternative, you could just `mmap` the entire file into memory. This will not improve or degrade performance in most cases, but it easier to work with.

int r, fdes;
struct stat st;
void *ptr;
size_t sz;

fdes = open(filename, O_RDONLY);
if (fdes < 0) abort();
r = fstat(fdes, &st);
if (r) abort();
if (st.st_size > (size_t) -1) abort(); // too big to map
sz = st.st_size;
ptr = mmap(NULL, sz, PROT_READ, MAP_SHARED, fdes, 0);
if (ptr == MAP_FAILED) abort();
close(fdes); // file no longer needed

// now, ptr has the data, sz has the data length
// you can use ordinary string functions

The advantage of using `mmap` is that your program won't run out of memory. On a 64-bit system, you can put the entire file into your address space at the same time (even a 10 GB file), and the system will automatically read new chunks as your program accesses the memory. The old chunks will be automatically discarded, and re-read if your program needs them again.

It's a very nice way to plow through large files.

Problem

I have a ASCII file where every line contains a record of variable length. For example ``` Record-1:15 characters Record-2:200 characters Record-3:500 characters ... ... Record-n: X characters ``` As the file sizes is about 10GB, i would like to read the record in chunks. Once read, i need to transform them, write them into another file in binary format. So, for reading, my first reaction was to create a char array such as ``` FILE *stream; char buffer[104857600]; //100 MB char array fread(buffer, sizeof(buffer), 104857600, stream); ``` - Is it correct to assume, that linux will issue one system call and fetch the entire 100MB? - As the records are separated by new line, i search for character by character for a new line character in the buffer and reconstruct each record. My question is that is this how i should read in chunks or is there a better alternative to read data in chunks and reconstitute each record? Is there an alternative way to read x number of variable sized lines from an ASCII file in one call ? Next during write, i do the same. I have a write char buffer, which i pass to fwrite to write a whole set of records in one call. ``` fwrite(buffer, sizeof(buffer), 104857600, stream); ``` UPDATE: If i setbuf(stream, buffer), where buffer is my 100MB char buffer, would fgets return from buffer or cause a disk IO?

Original source