What is a compact way to save a float32array to disk on node.js?

arrays, node.js, serialization

Solution

Finally I managed to write float32array to disk with nodejs and retrieve them on the browser, and I hope it will help you.

Write Float32Array to binary file in NodeJS

    var fs = require('fs');
    var wstream = fs.createWriteStream('data.dat');

    var data = new Float32Array([1.1,2.2,3.3,4.4,5.5]);

    //prepare the length of the buffer to 4 bytes per float
    var buffer = new Buffer(data.length*4);


    for(var i = 0; i < data.length; i++){
        //write the float in Little-Endian and move the offset
        buffer.writeFloatLE(data[i], i*4);
    }

    wstream.write(buffer);
    wstream.end();

Read the file and convert it to a Float32Array on a Browser

    var urlToFloatFile = 'data.dat';
    var request = new XMLHttpRequest();
    request.open('GET', urlToFloatFile, true);

    //specify the response type as arraybuffer
    request.responseType = 'arraybuffer';

    request.onload = function (msg) { 
        var yourFloatData = new Float32Array(this.response);
        console.log(yourFloatData);
    };
    request.send();

Thanks to @ben_a_adams from WebGL Dev List GGroup https://groups.google.com/forum/#!topic/webgl-dev-list/EbGUi_iSEx8 for the client side code

I've create a simple test to test roughly how much space a JSON serialization of a float array differs from a binary representation and the results are:

2.000.000 floating point values

7.8MB on a binary file

38.5MB on a JSON file

17.5 on a Gzipped JSON file

Problem

JSON.stringify is obviously not space-efficient. What is the most elegant way to serialize and store a float32array using Node.js? EDIT: People are closing the question for reasons such as being "opinion based" and "lack of an understanding of the problem". I seriously believe the first one was a missclick. For the second one, maybe this makes it more clear: ``` var fs = require("fs"); var len = 1000*1000*10; var big_array = new Float32Array(len); for (var i=0; i<len; ++i) big_array[i] = Math.random(); // OBVIOUSLY NOT SPACE EFFICIENT \/ fs.writeFileSync("big_array.json",JSON.stringify(big_array)); ``` It is not space efficient because you are representing numbers as strings, so an 8 bytes float will be using as much as ~20 utf8 chars, which is a waste. The question is: how to store the array in a space-efficient manner?

Original source