Pitest Mutation Testing to Improve Tests Quality

Good tests must fail: ensure meaningful testing with Pittest

Notepad with laptop, cup, pen
photo by ollie dale Feather unsplash

problem with standard test coverage

Writing unit tests is an essential part of the development process. We often use reporting tools such as jacoco To check test coverage. If we see that all the lines are green, then we are happy that we have covered everything.

However, full-line coverage can be achieved without any meaningful testing. For example, I can write tests without assertions or verifications. Sometimes, even if we cover all the cases, they may not be exhaustive enough.

How to improve the quality of testing?

mutation test There is a technique that checks whether each piece of code is meaningfully tested. It modifies the code in memory in different ways to produce different results. Then it is proved whether the tests will fail. Good tests must fail. We consider the test successful when the mutant (code modification) is killed.

The following figure shows how line and mutation coverage can differ:

PIT Report Coverage Example

In this article, I will introduce pittest (PIT) Mutation Testing Framework for Java with understandable examples. You will understand its benefits and learn how to integrate it into your project.

let’s get started!

I have created a simple java project to demonstrate the usage of PIT. The full code resides in my GitHub repo, which is linked under the References section at the end of the article.

project dependencies

It is a maven-based project. add pittest plugin to it pom.xml file:


org.pitest
pitest-maven
1.9.9

Note that pittest also works with other build tools such as Gradle.

We use JUnit as a testing framework:


org.junit.jupiter
junit
5.8.2
test

Note that PIT needs to know which test framework to use. so we have to provide pitest-junit5-plugin Plugin Dependencies:


org.pitest
pitest-maven
1.9.9


org.pitest
pitest-junit5-plugin
1.1.0


Standard Coverage Report

Let us first look at a simple example to understand the difference between line and mutation coverage.

Consider the following code:

    public static boolean isIsogram(String input) 
String[] splitString = input.split("");
Set set = new HashSet<>(Arrays.asList(splitString));
return input.length() == set.size();

An isogram is a word in which there is no repetition of letters. This code checks whether the given string is an isogram.

Let’s create a unit test:

@Test
void generateNumberFromRange_shouldReturnTrue()
var result = PitestDemo.isIsogram("chair");
System.out.println(result);

Note that I intentionally left out the assertion part to show you that the coverage will be green. I have run the jacoco report and see the following results:

Overall Jacoco Report
jacoco method line coverage

Tool-wise, we’ve got all the lines covered. it is not true.

Let’s add some assertions to make a more valid test:

@Test
void generateNumberFromRange_shouldReturnTrue()
var result = PitestDemo.isIsogram("chair");
Assertions.assertTrue(result);

It looks better, doesn’t it? However, the PIT report shows these results:

PIT Composite Report

In the next section, you’ll see how to configure PIT and generate reports.

coverage report with pittest

The basic configuration of the plugin might look like this:

 

com.mutation.demo.util.*


com.mutation.demo.util.*

  • targetClasses The property tells PIT where to inject the mutation.
  • targetTests Defines which unit tests to run.

PIT provides a rich choice of properties to define. To give you an idea, you can configure the following:

  • which classes or packages are out of scope (handy when you want to exclude third-party code)
  • what type of mutation to use (eg, negate boolean statements, remove void method calls, return void values, and so on.)
  • number of mutations per class
  • report settings

check it out Documentation for more information.

PIT automatically generates a report similar to Jacoco. We can see how our tests fared.

Execute the following maven command to run the test:

mvn test pitest:mutationCoverage

You should see a new folder under the target directory of the project called pit-reports,

open index.html file in your browser.

Now you see that PIT shows a slightly different result compared to Jacoco. Some mutants survived, which indicates that our tests need improvement. You can see what kind of mutants were created under the Mutation section. These are the default mutators. If you want to disable some or add new ones, configure it property in the plugin. is here list Of all the possible mutators.

Let’s write additional tests to cover the missing cases.

Reportedly, we don’t check the result when the method is wrong.

@Test
void generateNumberFromRange_shouldReturnFalse()
var result = PitestDemo.isIsogram("look");
Assertions.assertFalse(result);

Rerun the tests:

This time we covered all the cases!

Let us consider another example:

public static boolean isWithinRange(int number) 
return number <= 100 && number >= 10;

This method checks whether the provided number is between 10 and 100.

Here are the JUnit tests:

@ParameterizedTest
@ValueSource(ints = 15, 100)
void isWithinRange_shouldReturnTrue(int input)
var result = PitestDemo.isWithinRange(input);
Assertions.assertTrue(result);

@ParameterizedTest
@ValueSource(ints = 105, -150)
void isWithinRange_shouldReturnFalse(int input)
var result = PitestDemo.isWithinRange(input);
Assertions.assertFalse(result);

This time, we test with different inputs using @ParameterizedTest And @ValueSource Jupiter annotation.

This is the standard report:

jacoco report for method line coverage

PIT report looks like this:

Composite PIT Report

Detailed test case report:

Detailed Method Coverage Report

It’s almost good. We did not include a test to cover the lower limit, which is 10. Add values ​​to the values ​​list:

@ParameterizedTest
@ValueSource(ints = 10, 15, 100)
void isWithinRange_shouldReturnTrue(int input)
var result = PitestDemo.isWithinRange(input);
Assertions.assertTrue(result);

PIT Result:

Overall PIT report includes all

Great, everything is covered now!

This tutorial taught you how to use the Pittest plugin for mutation testing. This is beneficial to the quality of your unit tests.

As you noticed, the plugin is highly configurable. I encourage you to check the documentation for all the options to get the most out of it.

Note that performing mutation tests can be time consuming and expensive. It should be used with caution. Ideally, when we modify or add new classes.

I hope you have learned something new from this post. If you are interested in topics about testing, you might enjoy articles similar to mine:

Thanks for reading, and see you next time!

Leave a Reply