Sending messages from other languages to an IPython kernel

ipython, messaging, protocols, python, zeromq

Solution

This is a document that desparately needs to exist, but the implementation of the wire protocol is implemented in a single object, so it shouldn't be too hard to grok from there. The message spec doc you linked covers the application-level content of each field, but not how it's actually serialized over zeromq. Assuming you have a mesage, as described in that doc, the wire format is pretty simple. It is a multipart zeromq message of at least six parts:

- The leading message parts are the zeromq routing identities (zero-to-many)

- These are followed by a delimiter message, with the bytes `<IDS|MSG>`

- The hmac `digest` of the message (an empty string `''` if authentication is disabled)

- `header`

- `parent_header`

- `metadata`

- `content`

`header`, `parent_header`, `metadata`, and `content` are all described in the messaging doc - these are dictionaries, and serialized to bytes with whatever serialization is currently used. The default in IPython is utf8-encoded JSON, but arbitrary serialization is allowed (msgpack being the most common non-default). Not yet described in the docs is the `digest`, used for authentication. This is an MD5 HMAC Digest of the message. The key for the digest is found in the `key` field of the connection file. The 'message' used by the HMAC digest is the concatenation of the bytes of the serialized header, parent_header, metadata, and content, in the same order sent over the wire.

You can disable message signing by specifying the config value

Session.key = ''

to the IPython-related parts of your code, in which case the digest field will always be an empty string `''`. I would recommend doing this while getting started, so you can work out the more interesting parts of the implementation first.

Here are a sample execute request and its reply actually sent by IPython.

The request:

[
  <IDS|MSG>
  6ea6b213262402cc1ad3c1d3e342a9f6
  {"date":"2013-04-27T23:22:13.522049","username":"minrk","session":"5b03b89a-93c9-4113-bb85-17ba57233711","msg_id":"c6d0f85e-fc25-4f1e-84e1-3d706b615393","msg_type":"execute_request"}
  {}
  {}
  {"user_variables":[],"code":"1\n","silent":false,"allow_stdin":true,"store_history":true,"user_expressions":{}}
]

and its reply:

[
  5b03b89a-93c9-4113-bb85-17ba57233711
  <IDS|MSG>
  47d1052f6e8f333d18480938ca91719b
  {"date":"2013-04-27T23:22:13.528239","username":"kernel","session":"d7eb303b-d2d0-4723-aef2-738545a8da11","msg_id":"9ed1d332-398c-4132-b203-1e7bf8fed712","msg_type":"execute_reply"}
  {"date":"2013-04-27T23:22:13.522049","username":"minrk","session":"5b03b89a-93c9-4113-bb85-17ba57233711","msg_id":"c6d0f85e-fc25-4f1e-84e1-3d706b615393","msg_type":"execute_request"}
  {"dependencies_met":true,"engine":"645fb29f-37ab-40c9-bc01-b7fbfe3c2112","status":"ok","started":"2013-04-27T23:22:13.524114"}
  {"status":"ok","execution_count":2,"user_variables":{},"payload":[],"user_expressions":{}}
]

Problem

Does anyone have any experience of communicating with IPython kernels from outside of Python? If I were trying to send messages from a Python app to an IPython kernel, I'd use the `zmq.kernelmanager` API. As it is, I'll obviously need to write my own kernel manager in another language, but I can't find the information that I'm looking for about the low-level messaging protocols. Is there an official spec or a 'cheat sheet' that documents the structure of the actual messages that get sent over 0MQ? This page describes a higher-level protocol than what I'm looking for... Will I have to manually pull apart the implementation to find what I want?

Original source