I have red the article The Rise of Test Impact Analysis by Paul Hammant finding it very interesting.

I agree on the fact that developers might feel slowed down when tests execution on the development machine takes too long. Comprehensibly they might tend to not executing tests before commit, relying on tests running later on a continuous integration server. That fact might slowly drift developers off good Agile practices.

The article reports some «conventional strategies to shorten test automation» before describing «Test Impact Analysis». Test Impact Analysis (TIA) is a technique that helps determine which subset of tests execute for a given set of source code changes. The way by which the tests subset is determined starts with a code coverage or instrumentation phase, and finishes building a map which associate, for each entry, a test with the source code methods it will cover during its execution. TIA avoids executing all tests because it enables only the execution of the relevant ones.

Code Coverage as a technique to improve performance

I have found the article also interesting because it is not the first time I see code coverage being used as a technique to improve performance. In fact I have talked about Mutation Testing at the Ferrara LinuxDay 2015 and at the 72nd JUG Padua meeting.

Mutation Testing, originally proposed by Richard Lipton as a student in 1971, has recently became more popular thanks to availability of higher computing power. Mutation Testing exposes weaknesses in tests, therefore it is mainly a test quality tool. PIT, a state of the art mutation testing system, works by generating lots of mutants, where a mutant is a slightly modified version of your source code. Then PIT runs all tests against all mutants, and finally it checks which mutants has been killed or are still alive. The more mutants are killed the best your tests are, because they are able to detect changes in your source code.

The heavy computational part is «running all tests against all mutants». PIT lowers the computational cost in en elegant, yet simple, way: it leverages on line coverage produced by traditional code coverage tools. By knowing which test cover which source code, it runs for each mutants only tests which cover the mutated line of code.

If you like this image look at my presentation about mutation testing.

TIA support in Netbeans IDE

The Paul Hammant’s article ends with a little note about «TIA support in IDEs». My favourite IDE is Netbeans, and because I would like it supported TIA too, I have open a feature request. If you wanted TIA support in Netbeans too, go to the feature request and vote it please.