INTEGU uses cookies to personalize your experience and provide traceability for affiliate links. By using the website, you agree to these terms and conditions. To learn more see the privacy policy page.
Mutation Testing
What Is Mutation Testing – Beats 100% Code Coverage
Menu
What Is Mutation Testing?
Mutation testing is a way of testing software with the intention of creating bugs in the code and verifying that the test suites catch the bug. The need for mutation testing stems from the misleading information generated with normal code coverage. Code coverage accounts for the lines of codes, which have been passed through throughout all tests. However, taking the “happy path” through a method may produce 100% code coverage, but it will not cover all the alternative branches, which the method might also have available.
Cyclomatic Complexity & McCabe Testing Strategy
To cover all branches of a method’s logic flow, it is necessary to make a test for each if-statement, switch-case, logic operator (&&, ||), etc. Dependent on the number of branches a method consists of its cyclomatic complexity will rise or fall. A cyclomatic complexity of ten means that the method has ten possible branches.
The McCabe testing strategy describes that for each branch in the method, there should be a test method testing and verifying the output of that branch. Pitest does not actually refer to the McCabe testing strategy, but as we will see it ensure that, the developer writes code, which follows the same guiding principle. I will come back to Pitest later in the article.
To provide an example of how mutation testing work, let us take an example. Below we see a simple method, which consists of an if-statement and two different outputs dependent on the condition of the if-statement. To create a happy path test case, we would simply write one test method. However, this would not give us 100% code coverage. We therefore write two test cases. One for each of the return values. That is it, right?
Example (Before Mutation Testing)
Method
public static String methodToTest(int amountOfBeers) {
if (10 <= amountOfBeers) {
return "We got a party!";
}
return "Just a normal evening";
}
Test Cases
public void test_methodToTest_11_beers() {
assertEquals("We got a party!",
PartyChecker.methodToTest(11));
}
public void test_methodToTest_2_beers() {
assertEquals("Just a normal evening",
PartyChecker.methodToTest(2));
}
It is true that we now have 100% code coverage and that the method is somewhat tested. However, when we use mutation testing it will highlight how vulnerable our method actually is to changes. As mentioned in the beginning, a mutation test attempts to test the reliability of the test suites by introducing bugs into the code. Continuing with the example from above, we can try to imitate what a mutation testing tool would do to the method. Let us say that it simply removed the “=” (equals sign) from the if-statement. As it stand right now, none of the current test cases will fail. This means that we just changed the functionality of our 100% code covered method without a single sign of failure. This is of cause bad and clearly highlights how mutation testing can be a great utility for ensuring reliable code.
Example (After Mutation Testing)
Method
public static String methodToTest(int amountOfBeers) {
if (10 <= amountOfBeers) {
return "We got a party!";
}
return "Just a normal evening";
}
Test Cases
public void test_methodToTest_10_beers() {
assertEquals("We got a party!",
PartyChecker.methodToTest(10));
}
public void test_methodToTest_9_beers() {
assertEquals("Just a normal evening",
PartyChecker.methodToTest(9));
}
In the example above, the solution it simple: Write test cases with limiting arguments – e.g. 10 and 9. However, in other scenarios the method to be tested might have more branches and a more diverse output. (e.g. objects and exceptions) Using mutation testing in these cases will therefore be even more relevant and encourage the developer to start using more advanced testing validation methods than just asserts. (e.g. argument captures)
What Is Pitest?
Now that we know how mutation testing work, where do we then find it? I have had great experience using the Pitest tool, which can be found here (not an affiliate link). It provides quick on-demand analysis and comes with an intuitive user interface. This allows developers to evaluate their test cases live before having to push the code to branch with a CI (e.g. TeamCity) tool enabled. I highly encourage that you check it out, if you want to dive into the topic of mutation testing.
About
Hi, I'm the Author
My name is Daniel H. Jacobsen and I’m a dedicated and highly motivated software developer with a masters engineering degree within the field of ICT.
I have through many years of constantly learning and adapting to new challenges, gained a well-rounded understanding of what it takes to stay up to date with new technologies, tools and utilities.
The purpose of this blog is to share both my learnings and knowledge with other likeminded developers as well as illustrating how these topics can be taught in a different and alternative manner.
If you like the idea of that, I would encourage you to sign up for the newsletter.
Cheers! 🍺
