Slow Buffer.concat
buffer, node.js, performance
Solution
Rather than concatenating every time (which creates a new buffer each time), just keep an array of all of your buffers and concat at the end.
`Buffer.concat()` can take a whole list of buffers. Then it's done in one operation. https://nodejs.org/api/buffer.html#buffer_class_method_buffer_concat_list_totallength
Problem
When I read a 16MB file in pieces of 64Kb, and do `Buffer.concat` on each piece, the latter proves to be incredibly slow, takes a whole 4s to go through the lot. Is there a better way to concatenate a buffer in Node.js? Node.js version used: 7.10.0, under Windows 10 (both are 64-bit). This question is asked while researching the following issue: https://github.com/brianc/node-postgres/issues/1286, which affects a large audience. The PostgreSQL driver reads large `bytea` columns in chunks of 64Kb, and then concatenates them. We found out that calling `Buffer.concat` is the culprit behind a huge loss of performance in such examples.