Stream results in celery

celery, python

Solution

You did not specify many requirements and constraints. I'm going to assume you already have a redis instance somewhere.

What you can do is read the output from the other process line by line and publish it through redis:

Here's an example where you can `echo` data into a file `/tmp/foo` for testing:

import redis
redis_instance = redis.Redis()
p = subprocess.Popen(shlex.split("tail -f /tmp/foo"), stdout=subprocess.PIPE)

while True:
    line = p.stdout.readline()
    if line:
        redis_instance.publish('process log', line)
    else:
        break

In a separate process:

import redis

redis_instance = redis.Redis()
pubsub = redis_instance.pubsub()
pubsub.subscribe('process log')

while True:
    for message in pubsub.listen():
        print message  #  or use websockets to comunicate with a browser

If you want the process to end, you can e.g. send a "quit" after the celery task is done.

You can use different channels (the string in `subscribe`) to separate the output from different processes.

You can also store your log output in redis, if you want to,

redis_instance.rpush('process log', message)

and later retrieve it in full.

Problem

I'm trying to use celery to schedule and run tasks on a fleet of servers. Each task is somewhat long running (few hours), and involves using subprocess to call a certain program with the given inputs. This program produces a lot of output both in stdout and stderr. Is there some way to show the output produced by the program to the client in near real time? Stream the output, so that the client can watch the output spewed by the task running on the server without logging into the server?

Original source