Memory leak using tornado's gen.engine

Question

I have code which, in a simplified form, looks like this: ``` from tornado import gen, httpclient, ioloop io_loop = ioloop.IOLoop.instance() client = httpclient.AsyncHTTPClient(io_loop=io_loop) @gen.engine def go_for_it(): while True: r = yield gen.Task(fetch) @gen.engine def fetch(callback): response = yield gen.Task(client.fetch, 'http://localhost:8888/') callback(response) io_loop.add_callback(go_for_it) io_loop.start() ``` When I run it the memory footprint keeps increasing over time more or less linearly. If, however, I remove the `gen.engine` nesting: ``` @gen.engine def go_for_it(): while True: r = yield gen.Task(client.fetch, 'http://localhost:8888/') ``` memory usage remains constant. I've managed to reproduce the issue with different versions of tornado 2, on both Mac OS X and Linux. Any ideas what might be the cause of this problem?

Accepted Answer

Digging into it with the help of objgraph package, it looks like the code leaks ExceptionStackContexts. These are created by gen.engine to handle function exceptions, they should be cleared up but clearly you found a bug.

My best guess is that the there is a reference left somewhere to one.

EDIT - The Tornado Team (Ben) has found a fix and it will be in a future release. https://github.com/facebook/tornado/commit/bff07405549a6eb173a4cfc9bbc3fc7c6da5cdd7

Memory leak using tornado's gen.engine

Solution

Problem