async parallel request - partial render

asynchronous, express, node.js

Solution

what you want is facebook's bigpipe: https://www.facebook.com/note.php?note_id=389414033919. fortunately, this is easy with nodejs because streaming is built in. unfortunately, template systems are bad at this because async templates are a pain in the butt. however, this is much better than doing any additional AJAX requests.

basic idea is you first send a layout:

res.render('layout.ejs', function (err, html) {
  if (err) return next(err)

  res.setHeader('Content-Type', 'text/html; charset=utf-8')
  res.write(html.replace('</body></html>', ''))

  // Ends the response. 
  // `writePartials` should not return anything in the callback!
  writePartials(res.end.bind(res, '</body></html>'))
})

you can't send `</body></html>` because your document isn't finished. then `writePartials` would be a bunch of async functions (partials or pagelets) executed in parallel.

function writePartials(callback) {
  async.parallel([partial1, partial2, partial3], callback)
})

Note: since you've already written a response, there's not much you can do with errors except log them.

What each partial will do is send inline javascript to the client. For example, the layout can have `.stream`, and the pagelet will replace `.stream`'s `innerHTML` upon arrival, or when "the callback finishes".

function partialStream(callback) {
  res.render('stream.partial.ejs', function (err, html) {
    // Don't return the error in the callback
    // You may want to display an error message or something instead
    if (err) {
      console.error(err.stack)
      callback()
      return
    }

    res.write('<script>document.querySelector(".stream").innerHTML = ' + 
      JSON.stringify(html) + ';</script>')
    callback()
  })
})

Personally, I have `.stream.placeholder` and replace it with a new `.stream` element. The reason is I basically do `.placeholder, .placeholder ~ * {display: none}` so things don't jump around the page. However, this requires a DIY front-end framework since suddenly the JS gets more complciated.

There, your response is now streaming. Only requirement is that the client supports Javascript.

Problem

What is the proper way to partially render a view following an async parallel request? Currently I am doing the following ``` // an example using an object instead of an array async.parallel({ one: function(callback){ setTimeout(function(){ callback(null, 1); // can I partially merge the results and render here? }, 200); }, two: function(callback){ setTimeout(function(){ callback(null, 2); // can I partially merge the results and render here? }, 100); } }, function(err, results) { // results is now equals to: {one: 1, two: 2} // merge the results and render a view res.render('mypage.ejs', { title: 'Results'}); }); ``` It is basically working fine, but, if I have a `function1, function2, ..., functionN` the view will be rendered only when the slowest function will have completed. I would like to find the proper way to be able to render the view as soon as the first function is returning to minimise the user delay, and add the results of the function as soon as they are available.

Original source