An introduction to mutation testing and PIT

This post was published on February 9, 2016

This blog post will cover the basics of the concept of mutation testing. I have been made aware of mutation testing only recently, but I have discovered it’s a very powerful and interesting technique for:

  • analysis and improvement of unit tests
  • detection of dead code in your application

Two things that are always worth taking a look at if you ask me. I will illustrate the mutation testing concept using a tool called PIT and a simple piece of code and accompanying set of unit tests.

What is mutation testing?
From Wikipedia:

Mutation testing is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill.

In other words, mutation testing is a technique that allows you to evaluate not only the percentage of code that is executed when running your tests (i.e., code coverage), but also the ability of your tests to detect any defects in the executed code. This makes mutation testing a very powerful and very useful technique I think anyone involved in software development and testing should at least be aware of.

Introducing PIT
I will try and illustrate the power of mutation testing using PIT, a Java mutation test tool which can be downloaded here. I chose PIT over other available mutation test tools mainly because of its ease of installation and use.

Assuming you’re also using Maven, you can configure your Java project for mutation testing using PIT by adding the following to your pom.xml:

<build>
	<plugins>
		<plugin>
			<groupId>org.pitest</groupId>
			<artifactId>pitest-maven</artifactId>
			<version>PIT-VERSION</version>
			<configuration>
				<targetClasses>
					<param>package.root.containing.classes.to.mutate*</param>
				</targetClasses>
				<targetTests>
					<param>package.root.containing.test.classes*</param>
				</targetTests>
			</configuration>
		</plugin>
	</plugins>
</build>

Simply replace the package locators with those appropriate for your project and be sure not to forget the asterisk at the end. Also replace PIT-VERSION with the PIT version you want to use (the latest is 1.1.4 at the moment of writing this blog post) and you’re good to go.

The code class and test class to be subjected to mutation testing
I created a very simple Calculator class that, you guessed it, performs simple arithmetic on integers. My calculator only does addition, subtraction and power calculations:

public class Calculator {

	int valueDisplayed;

	public Calculator() {
		this.valueDisplayed = 0;
	}
	
	public Calculator(int initialValue) {
		this.valueDisplayed = initialValue;
	}

	public void add(int x) {
		this.valueDisplayed += x;
	}
	
	public void subtract(int x) {
		this.valueDisplayed -= x;
	}
	
	public void power(int x) {
		this.valueDisplayed = (int) Math.pow(this.valueDisplayed, x);
	}

	public int getResult() {
		return this.valueDisplayed;
	}
	
	public void set(int x) {
		this.valueDisplayed = x;
	}

	public boolean setConditional(int x, boolean yesOrNo) {
		if(yesOrNo) {
			set(x);
			return true;
		} else {
			return false;
		}
	}
}

To test the calculator, I have created a couple of TestNG unit tests that call the various methods my calculator supports. Note that PIT supports both JUnit and TestNG.

public class CalculatorTest {
	
	@Test
	public void testAddition() {
		
		Calculator calculator = new Calculator();
		calculator.add(2);
		Assert.assertEquals(calculator.getResult(), 2);
	}
	
	@Test
	public void testPower() {
		
		Calculator calculator = new Calculator(2);
		calculator.power(3);
		Assert.assertEquals(calculator.getResult(), 8);
	}
	
	@Test
	public void testConditionalSetTrue() {
		
		Calculator calculator = new Calculator();
		Assert.assertEquals(calculator.setConditional(2, true), true);
	}
	
	@Test
	public void testConditionalSetFalse() {
		
		Calculator calculator = new Calculator();
		Assert.assertEquals(calculator.setConditional(3, false), false);
	}
}

To illustrate the capabilities of PIT and mutation testing in general, I ‘forgot’ to include a test for the subtract() method. Also, I created what is known as a ‘weak test’: a test that passes but doesn’t check whether all code is actually called (in this case, no check is done to see whether set() is called when calling setConditional()). Now, when we run PIT on our code and test classes using:

mvn org.pitest:pitest-maven:mutationCoverage

an HTML report is generated displaying our mutation test results:

The report as generated by PIT

When we drill down to our Calculator class we can see the modifications that have been made by PIT and the effect it had on our tests:

Class-level details of the PIT mutation test results

This clearly shows that our unit test suite has room for improvement:

  • The fact that subtract() is never called in our test suite (i.e., our code coverage can be improved) is detected
  • The fact that the call to set() can be removed from the code without our test results being affected (i.e., our tests are lacking defect detection power) is detected

These holes in our test coverage and test effectiveness might go unnoticed for a long time, especially since all tests pass when run using TestNG. This goes especially for the second flaw as a regular code coverage tool would not pick this up: the call to set() is made after all, but it does not have any effect on the outcome of our tests!

Additional PIT features
The PIT documentation discusses a lot of features that make your mutation testing efforts even more powerful. You can configure the set of mutators used to tailor the result set to your needs, you can use mutation filters to filter out any unwanted results, and much more. However, even in the default configuration, using PIT (or potentially any other mutation testing tool as listed here will tell you a lot about the quality of your unit testing efforts.

Removing dead code from your codebase based on mutation test results
Apart from evaluating the quality of your unit tests, mutation test results can also give you insight into which parts of your application code are never executed (dead code). Consider the call to the set() method in the example above. The mutation test results indicated that this call could be removed without the results of the unit test being altered. Now, in our case it is pretty obvious that this indicates a lack of coverage in our unit tests (if you want to set the Calculator value, you’d better call the set() method), but it isn’t hard to imagine a situation where such a method call can be removed without any further consequences. In this case, the results of the mutation tests will point you to potentially dead code that might be a candidate for refactoring or removal. Thanks go to Markus Schirp for pointing out this huge advantage of mutation testing to me on Twitter.

Example project
The Maven project that was used to generate the results demonstrated in this post can be downloaded here. You can simply import this project and run

mvn org.pitest:pitest-maven:mutationCoverage

to recreate my test results and review the generated report. This will serve as a good starting point for your further exploration of the power and possibilities of mutation testing.

"