2. Blog
  3. Going Beyond Code Coverage With Mutation Testing

Going Beyond Code Coverage With Mutation Testing

13-04-2025 door Roy de Kleijn

Automated tests give us confidence. But how do we know our tests are actually good at catching real bugs? High code coverage looks nice on a report, but 100% coverage doesn’t mean your tests are effective.

This is where mutation testing comes in.

Instead of checking which lines are executed, mutation testing introduces small changes (mutants) to your code. Then, it checks whether your existing tests fail. If a test passes even after a logical change, that means your test didn’t catch a potential bug.

Let’s look at a real example using a simple Java coffee machine project. Coffee Machine Simulator

Project Overview

We have a CoffeeMachine class and a set of tests built with JUnit 5. One key part of the machine is a Reservoir, used to store water, beans, and milk. The reservoir ensures you can't brew more coffee than you have resources for.

Here’s one of the classes:

public abstract class AbstractReservoir implements Reservoir {
    protected int level;
    protected final int capacity;

    public AbstractReservoir(int capacity) {
        this.capacity = capacity;
        this.level = capacity;
    }

    @Override
    public void use(int amount) {
        if (amount > level) {
            throw new IllegalArgumentException("Not enough resources!");
        }
        level -= amount;
    }

    @Override
    public void refill(int amount) {
        level = Math.min(level + amount, capacity);
    }

    @Override
    public int getLevel() {
        return level;
    }
}

We’ve got tests for use(), refill(), and different brewing scenarios like lattes, espressos, etc.

Running Mutation Tests with PIT

To run mutation testing, we use PIT (Pitest). After setting it up in our Maven pom.xml, we run:

mvn test-compile org.pitest:pitest-maven:mutationCoverage

It generates an HTML report showing which mutations were detected by tests and which survived.

The Problem PIT Found

Here’s one of the mutations PIT applied to the use() method: Original:

if (amount > level) {
    throw new IllegalArgumentException("Not enough resources!");
}

Mutant:

if (amount >= level) {
    throw new IllegalArgumentException("Not enough resources!");
}

This changes the logic subtly. With the mutant, trying to use exactly the remaining amount would now fail. And our tests didn’t catch it.

PIT marked this mutant as SURVIVED, meaning no test failed because of this change.

PIT Report

That’s a gap in the tests.

The Fix: Add a Boundary Test

To catch this mutation, we add a test where the amount used is exactly equal to the current level:

@Test
public void testReservoir_UseExactAvailableAmount() {
    BeanReservoir reservoir = new BeanReservoir(50);
    reservoir.use(50); // Should succeed
    assertEquals(0, reservoir.getLevel());
}

This test will fail if the mutant (>=) is applied, which means the mutant will be killed during the next mutation test run.

After rerunning PIT:

mvn org.pitest:pitest-maven:mutationCoverage

✅ The mutation is caught. Test effectiveness just went up.

PIT Report

Final Thoughts

Code coverage tells you what code is executed, not whether it’s actually tested well. Mutation testing fills that gap by actively trying to break your tests.

In our case, the tests looked complete, but mutation testing found a real logic edge case that was missed.

If you're already testing with JUnit and Maven, adding PIT is simple and powerful. And it’s a great way to ensure your tests are pulling their weight.

Delen:

Categories