node - send large JSON over net socket
node.js
Solution
- You can use try...catch every time to see if it is a valid json. Not very good performance though.
- You can calculate size of your json on sending side and send it before JSON.
You can append a boundary string that's unlikely be in JSON. Your \u0000 - yes, it seems to be a legit way. But most popular choice is newline.
You can use external libraries like dnode which should already do something I mentioned before. I'd recommend trying that. Really.
One of the failures of this model is if the receiver is connected to multiple sockets, and each socket is sending large JSON files.
Use different buffers for every socket. No problem here.
Problem
The problem is that sending large serialized JSON (over 16,000 characters) over a net socket gets split into chunks. Each chunk fires the `data` event on the receiving end. So simply running `JSON.parse()` on the incoming data may fail with `SyntaxError: Unexpected end of input`. The work around I've managed to come up with so far is to append a null character (`'\u0000'`) to the end of the serialized JSON, and check for that on the receiving end. Here is an example: ``` var partialData = ''; client.on( 'data', function( data ) { data = data.toString(); if ( data.charCodeAt( data.length - 1 ) !== 0 ) { partialData += data; // if data is incomplete then no need to proceed return; } else { // append all but the null character to the existing partial data partialData += data.substr( 0, data.length - 1 ); } // pass parsed data to some function for processing workWithData( JSON.parse( partialData )); // reset partialData for next data transfer partialData = ''; }); ``` One of the failures of this model is if the receiver is connected to multiple sockets, and each socket is sending large JSON files. The reason I'm doing this is because I need to pass data between two processes running on the same box, and I prefer not to use a port. Hence using a net socket. So there would be two questions: First, is there a better way to quickly pass large JSON data between two Node.js processes? Second, if this is the best way then how can I better handle the case where the serialized JSON is being split into chunks when sent?