How to configure Luigi task retry correctly?
luigi, python
Solution
This is an old issue of Luigi - where successful retried tasks were not marked as such when failed and then succeeded on retry: https://github.com/spotify/luigi/issues/1932
It was fixed in version 2.7.2: https://github.com/spotify/luigi/releases/tag/2.7.2
I suggest you upgrade to the latest Luigi version, i.e. by running `pip install -U luigi`.
Problem
I am trying to configure Luigi's retry mechanism so that failed tasks will be retried a few times. However, while the task is retried successfully, Luigi exits unsuccessfully: ``` ===== Luigi Execution Summary ===== Scheduled 3 tasks of which: * 2 ran successfully: - 1 FailOnceThenSucceed(path=/tmp/job-id-18.subtask) - 1 MasterTask(path=/tmp/job-id-18) * 1 failed: - 1 FailOnceThenSucceed(path=/tmp/job-id-18.subtask) This progress looks :( because there were failed tasks ``` So the question is: how do I configure Luigi (I have installed version 2.3.3 with pip install) so that when a task fails once, but is then retried with success, then Luigi will exit successfully with `This progress looks :)` instead of fail with `This progress looks :(`? Here is a minimal scheduler and worker config I've come up with, as well as tasks to demonstrate the behavior: ``` [scheduler] retry_count = 3 retry-delay = 1 [worker] keep_alive=true ``` mytasks.py: ``` import luigi class FailOnceThenSucceed(luigi.Task): path = luigi.Parameter() def output(self): return luigi.LocalTarget(self.path) def run(self): failmarker = luigi.LocalTarget(self.path + ".hasfailedonce") if failmarker.exists(): with self.output().open('w') as target: target.write('OK') else: with failmarker.open('w') as marker: marker.write('Failed') raise RuntimeError("Failed once") class MasterTask(luigi.Task): path = luigi.Parameter() def requires(self): return FailOnceThenSucceed(path=self.path + '.subtask') def output(self): return luigi.LocalTarget(self.path) def run(self): with self.output().open('w') as target: target.write('OK') ``` Example execution: `PYTHONPATH=. luigi --module mytasks MasterTask --workers=2 --path='/tmp/job-id-18'`