best way to append text often in MATLAB

matlab

Solution

There is not an obvious best answer to this, at least for me. Some options are:

- Exactly what you are doing, incrementally appending to a string for each iteration

- Intelligently growing your accumulation string to reduce the number of reallocations (the core of the @macduff answer)

- Using a cell array of strings, and intelligently reallocating that. (I'm pretty sure) that this only forces a reallocation of pointers, not a full reallocation of the string contents.

- Using some Java magic to handle string accumulation. The Java library has many useful features (such as a StringBuilder class), but the Matlab-Java interface is slow.

- Incrementally writing directly to the file (I know that you have removed this from consideration in the question, but it is still a useful baseline.)

My intuition indicates that the performance order would be:

- Best: (2 or 3)

- Mid: (4 or 5)

- Worst: (1)

but it's not obvious.

Fortunately it's easy to test. An implementation of all 5 options (plus some testing wrappers) is included in the large test block below. The results on my computer (a pretty good computer with an SSD, results may vary) are below (spaces added to the code output for formatting):

-------------Start of file write speed tests.  (nLines = 1)------------
Time for BaseLine operation:                0.001540 sec
Time for AutoAllocate operation:            0.001264 sec
Time for AutoAllocateCell operation:        0.003492 sec
Time for JavaStringBuilder operation:       0.001395 sec
Time for IncrementalWriteToFile operation:  0.001057 sec
-------------Start of file write speed tests.  (nLines = 100)------------
Time for BaseLine operation:                0.011909 sec
Time for AutoAllocate operation:            0.014067 sec
Time for AutoAllocateCell operation:        0.011517 sec
Time for JavaStringBuilder operation:       0.021291 sec
Time for IncrementalWriteToFile operation:  0.016213 sec
-------------Start of file write speed tests.  (nLines = 10000)------------
Time for BaseLine operation:                3.778957 sec
Time for AutoAllocate operation:            1.048480 sec
Time for AutoAllocateCell operation:        0.856269 sec
Time for JavaStringBuilder operation:       1.657038 sec
Time for IncrementalWriteToFile operation:  1.254080 sec
-------------Start of file write speed tests.  (nLines = 100000)------------
Time for BaseLine operation:              358.312820 sec
Time for AutoAllocate operation:           10.349529 sec
Time for AutoAllocateCell operation:        8.539117 sec
Time for JavaStringBuilder operation:      16.520797 sec
Time for IncrementalWriteToFile operation: 12.259307 sec

So, if you are using "100's" of lines, it may not matter very much; do whatever works. If you know performance matters, then I would use the "AutoAllocateCell" option. It's pretty easy code (see below). If you do not have enough memory to store the whole file in memory at once, I would use the "AutoAllocateCell" option with a periodic flush to file.

Test code:

%Setup
cd(tempdir);
createLineLine = @(n, s) sprintf('[%04d]  %s\n', n, s);
createRandomLine = @(n) createLineLine(n, char(randi([65 122],[1, round(rand*100)])));

for nLines = [1 100 10000 100000]        
    fprintf(1, ['-------------Start of file write speed tests.  (nLines = ' num2str(nLines) ')------------\n']);

    %% Baseline -----------------------------
    strName = 'BaseLine';
    rng(28375213)
    tic;

    str = [];
    for ix = 1:nLines;
        str = [str createRandomLine(ix)];
    end

    fid = fopen(['WriteTest_' strName],'w');
    fprintf(fid, '%s', str);
    fclose(fid);

    fprintf(1, 'Time for %s operation: %f sec\n', strName, toc);

    %% AutoAllocated string -----------------------------
    strName = 'AutoAllocate';
    rng(28375213)
    tic;

    str = blanks(256);
    ixLastValid = 0;
    for ix = 1:nLines;
        strNewLine = createRandomLine(ix);
        while (ixLastValid+length(strNewLine)) > length(str)
            str(end*2) = ' ';  %Doubles length of string
        end
        str(ixLastValid + (1:length(strNewLine))) = strNewLine;
        ixLastValid = ixLastValid+length(strNewLine);
    end

    fid = fopen(['WriteTest_' strName],'w');
    fprintf(fid, '%s', str(1:ixLastValid));
    fclose(fid);

    fprintf(1, 'Time for %s operation: %f sec\n', strName, toc);

    %% AutoAllocated cell array -----------------------------
    strName = 'AutoAllocateCell';
    rng(28375213)
    tic;

    strs = cell(256,1);
    ixLastValid = 0;
    for ix = 1:nLines;
        if ix>length(strs);
            strs{end*2} = {};  %Doubles cell array size;
        end
        strs{ix} = createRandomLine(ix);
        ixLastValid = ixLastValid + 1;
    end

    fid = fopen(['WriteTest_' strName],'w');
    fprintf(fid, '%s', strs{1:ixLastValid});
    fclose(fid);

    fprintf(1, 'Time for %s operation: %f sec\n', strName, toc);

    %% Java string builder -----------------------------
    strName = 'JavaStringBuilder';
    rng(28375213)
    tic;

    sBuilder = java.lang.StringBuilder;
    for ix = 1:nLines;
        sBuilder.append(createRandomLine(ix));
    end

    fid = fopen(['WriteTest_' strName],'w');
    fprintf(fid, '%s', char(sBuilder.toString()));
    fclose(fid);

    fprintf(1, 'Time for %s operation: %f sec\n', strName, toc);

    %% Incremental write to file -----------------------------
    strName = 'IncrementalWriteToFile';
    rng(28375213)
    tic;

    fid = fopen(['WriteTest_' strName],'w');
    for ix = 1:nLines;
        fprintf(fid, '%s', createRandomLine(ix));
    end
    fclose(fid);

    fprintf(1, 'Time for %s operation: %f sec\n', strName, toc);
end

Problem

Original source