Job orchestration
orchestration, python, scheduled-tasks
Solution
Probably to late, but I wanted to mention Job runner for possibly other people arriving at this question.
From their GitHub repository README.rst:
Job-Runner is a crontab like tool, with a nice web-frontend for administration and (live) monitoring the current status.
Features:
- Schedule recurring jobs
- Chaining of jobs
- Load-balance workers by putting them in a pool
- Schedule jobs to run on all workers within a pool
- Live dashboard (with option to kill runs and ad-hoc scheduling)
- Multiple projects and per-project permission management
Problem
I am currently redoing all our database orchestration jobs (ETL, backups, daily tasks, report compilation, etc.) To do that, I would need a task/job orchestrator where I can define tasks dependency, time based tasks, async tasks, etc. I was looking at celery and Flow Based Programming technologies but I am not sure these are good for my use case. I am looking more at a framework that would support all these things out of the box. Any suggestions?