Find the commit that broke the test without running every test on every commit
I am trying to find a tool that can solve the following problem:
Our entire test suite takes several hours, which often makes it difficult or at least very time consuming to figure out which commit has imposed a particular test, since there can be 50 to 200 between test runs. At any given time, there is only very little broken tests, so repeating only broken tests is very fast compared to running the entire test suite.
Is there a tool for example. a continuous integration server that can rerun failed tests with two revisions between the last version where the test was ok and the first version where the test was not OK, and therefore automatically detects on which particular commit the test switched from successful to broken.
For example:
Testing A and B are normal 100. Test A and B are broken in version 200.
The tool should now run both tests with revision 150. Then, if, for example, test A was broken and test B was fine at revision 150, it could continue checking test A with revision 125 and test B with revision 175, and so on. until every broken test can be explained by any particular fixer.
For one test, I could probably hack something along with git bisect. But for a few failed tests, this is probably not enough, as we need to search both ways for many changes.
source to share
git
do you use git or mercurial(see: What is Mercurial bisect for? )?
Suppose version 2.6.18 of your project worked, but the version on "master" worked. Sometimes the best way to find the cause of such a regression is to do a brute force search in the project history to find the specific commit that caused the problem. The git bisect command can help you do this: [...]
From: Troubleshooting - git Bisect .
Essentially you mark the two versions as start and end, and you end up:
$ git bisect
Then run the test and depending on whether it passed or suspended the call
$ git bisect good
or
$ git bisect bad
respectively. git will do a binary search, always cutting the remaining changes in half. I think you can script it easily. If you are usingsvngit can import the whole repository easily.
Create a single revision at a time
This is not an answer, but a tip - just check every commit! Today, you can easily cluster your CI servers, and a server farm is not that hard to check all the commits.
source to share