Best way to map a generated list to a task in celery

Question

I am looking for some advice as to the best way to map a list generated from a task to another task in celery. Let's say I have a task called `parse`, which parses a PDF document and outputs a list of pages. Each page then needs to be individually passed to another task called `feed`. This all needs to go inside a task called `process` So, one way I could do that is this: ``` @celery.task def process: pages = parse.s(path_to_pdf).get() feed.map(pages) ``` Of course, that is not a good idea because I am calling `get()` inside a task. Additionally this is inefficient, since my `parse` task is wrapped around a generator function and is able to yield pages, which means that it should be possible to queue the first page for feeding before the last page has been yielded by the parser. Another possibility is to do this: ``` @celery.task def process: for page in parse.s(path_to_pdf).get(): feed.delay(page) ``` That example still involves calling `get()` inside a task though. Additionally, thi…

Accepted Answer

This is probably far too late to be of use to you, but you probably want to use a task chain:

```
@celery.task
def process():
    return chain(parse.s(), feed_map.s())

@celery.task
def feed_map(pages):
    return feed.map(pages)
```

if you have some final task, say `final`, you could do this:

```
@celery.task
def feed_map(pages):
    return chord(feed.map.s(page) for page in pages, final.s)
```

Best way to map a generated list to a task in celery

Solution

Problem