Finding @ Ignore'd tests currently passing
For a decent sized open source project with developers coming and going, someone might fix the bug without realizing that someone else has disabled the unit test some time ago (a la @Ignore). We would like to find passed tests that are disabled, so we can enable them and update the bug tracking, CC list, and everything else adrift.
What is the best way to periodically run all @ Ignore'd tests and identify the ones that are currently passing? We are using Java 1.6 with JUnit4, build our project with ant and go to gradle. We are using Jenkins for CI.
Some ideas:
-
Forever replace all our @Ignore annotations with conditional ignore http://www.codeaffine.com/2013/11/18/a-junit-rule-to-conditionally-ignore-tests/
-
Run a custom JUnit4 class that modifies @Ignore's behavior. stackoverflow question
-
Temporarily comment out all @Ignore annotations so they run. However, we need a way to nullify errors.
source to share
Sorry this is not a solution, but rather another alternative that worked for me: My key point was not to modify existing (1000) unit tests. Thus, the general code does not change. No new annotations, certainly not temporary.
What I did was override @Ignore's JUnit detection and made it conditional, via class path prefixes: check separate control file if this test / class is specified or disabled. This is based on package / FQCN / method names and Regexp patterns. If it's enabled, run it even though it still has @Ignore in the unchanged JUNit Test original source.
Record the result, modify the control file. Rinse and repeat.
source to share