nodejs/express and binary data in POST

binary, express, node.js

Solution

It turned out to be an encoding issue. Everything seems to default to 'utf8', but in this case it needs to be set to 'binary'.

For example:

var buf = new Buffer(body.toString('binary'),'binary');

Equally as important, I needed to set nodejs multiparty parser's encoding to 'binary'. See: https://github.com/superjoe30/node-multiparty/issues/37

Problem

I'm trying to send binary data to an express app. It works fine, as long as my values are smaller than 0x80. If a single value is 0x80 or greater, it messes up the whole buffer. Express Handler: ``` binary = require('binary'); exports.api = function(req, res){ var body = req.body.name; var buf = new Buffer(body,'binary'); console.log('body',req.body); console.log('req body len', body.length); console.log('buf len', buf.length); var g = binary.parse(buf) .word16bu('a') // unsigned 16-bit big-endian value .word16bu('b').vars console.log('g.a', g.a); console.log('g.b', g.b); res.send("respond with a resource"); }; ``` Python client (Content-Type: application/x-www-form-urlencoded): ``` import requests from struct import pack # send two unsigned shorts (16-bits each). requests.post('http://localhost:3000/api', data={'name':pack('!HH',1,2)}) ``` Express output when data = 1,2. This is what I'd expect. ``` body { name: '\u0000\u0001\u0000\u0002' } req body len 4 buf len 4 g.a 1 g.b 2 POST /api 200 1ms - 23b ``` Express output when data = 1,0xFF. It's interesting to note that 9520 is actually 0x25 0x30 in hex, which corresponds to "%0" in ASCII. Yes, it appears to be parsing the '%00%01...' string. I wish I knew how to prevent this!!! ``` body { name: '%00%01%00%FF' } req body len 12 buf len 12 g.a 9520 g.b 12325 POST /api 200 2ms - 23b ```

Original source