10 fold size increase when reading file into struct

c++, struct

Solution

I tested your code with an input file that has `1886850` lines of text, with a size of `105M`.

With your code, the memory consumption was about 2.5G.

Then, I started modifying how data is stored.

First test:

Change `datStr` to:

struct datStr{
    vector<string> colNames;
    vector<string> lines;
};

This reduced the memory consumption to `206M`. That's more than 10 fold reduction in size. It's clear that the penalty for using

vector<vector<string>> data;

is rather stiff.

Second test:

Change `datStr` to:

struct datStr{
    vector<string> colNames;
    vector<string> lines;
    vector<vector<string::size_type>> indices;
};

with `indices` keeping track of where the tokens in `lines` start. You can extract the tokens from each line by using `lines` and `indices`.

With this change, the memory consumption went to `543MB` but sill is five times smaller than the original.

Third test

Change `dataStr` to:

struct datStr{
    vector<string> colNames;
    vector<string> lines;
    vector<vector<unsigned int>> indices;
};

With this change, the memory consumption came down to `455MB`. This should work if you don't expect your lines to be longer or equal to `UINT_MAX`.

Fourth Test

Change `dataStr` to:

struct datStr{
    vector<string> colNames;
    vector<string> lines;
    vector<vector<unsigned short>> indices;
};

With this change, the memory consumption came down to `278MB`. This should work if you don't expect your lines to be longer or equal to `USHRT_MAX`. For this case, the overhead of `indices` is really small, only `72MB`.

Here's the modified code I used for my tests.

#include<iomanip>
#include<stdio.h>
#include<stdlib.h>
#include<iostream>
#include<fstream>
#include<string.h>
#include<sstream>
#include<math.h>
#include<vector>
#include<algorithm>
#include<array>
#include<ctime>
// #include<boost/algorithm/string.hpp>
using namespace std;

struct datStr{
    vector<string> colNames;
    vector<string> lines;
    vector<vector<unsigned short>> data;
};

void split(vector<unsigned short>& rowStr, string const& line)
{
   string::size_type begin = 0;
   string::size_type end = line.size();
   string::size_type iter = begin;
   while ( iter != end)
   {
      ++iter;
      if ( line[iter] == ',' )
      {
         rowStr.push_back(static_cast<unsigned short>(begin));
         ++iter;
         begin = iter;
      }
   }
   if (begin != end )
   {
      rowStr.push_back(static_cast<unsigned short>(begin));
   }
}

datStr readBoost(string fileName)
{
   datStr ds;
   ifstream inFile;
   inFile.open(fileName);
   string line;
   getline(inFile, line);
   vector<string> colNames;
   stringstream ss(line);
   string item;
   int i = 0;
   vector<int> colTypeInt;
   while(getline(ss, item, ','))
   {
      item.erase( remove( item.begin(), item.end(), ' ' ), item.end() );
      ds.colNames.push_back(item);
   }

   int itr = 0;
   while(getline(inFile, line))
   {
      ds.lines.push_back(line);
      vector<unsigned short> rowStr;
      split(rowStr, line);
      ds.data.push_back(rowStr);
   }
}

int main(int argc, char** argv)
{
   datStr ds = readBoost(argv[1]);
   while(true)
   {
   }
}

Problem

Original source