Failing tests allowed in Bamboo?
bamboo, continuous-integration, junit
Solution
It seems that Bamboo does not support this use case. There are some plugins that try to deal with this.
What I finally did was to create an intermediate script that parses the test results (JUnit XML). There are three scenarios here:
- all test pass: the XML is left as is
- there are too many failing tests: the XML is left as is (the build will fail)
- there are failing tests, but their number does not exceed the set threshold: the XML contents are substituted with "passing" contents so that Bamboo does not fail the build
Problem
Is it possible to somehow set a threshold for JUnit test cases that would stop Bamboo from failing a build when there are failing tests? I know about the 'Quarantine' option, but it is not exactly what I am looking for. I do not want to have to manually move the test cases from the quarantined set. We have a TDD environment, where the test cases are based on a large set of external, annotated data, and we want to build up the coverage. I want to see the actual results (e.g.: 43% pass) and tell Bamboo to succeed the build e.g. when the results are >40%. By the way, if I stick to Quarantine, do I really have to pick them one by one? There are 10,000! :D