Coverage with Jacoco and Sonarqube

By Gerald Mücke | December 4, 2018

Coverage with Jacoco and Sonarqube

In most projects I have worked in, Jacoco was used as tool to determine code coverage. The configuration is fairly easy as it plugs into the JVM that runs the tests using an agent that tracks the invocations. In maven, this JVM is forked by the surefire plugin and the parameters are auto generated. The setup is well documented so in this blog post I want to shed some lights on the internals of Jacoco and Sonarqube and how both calculate their coverage metrics. I did some code digging, and I’d like to share my insights. The following information is a compilation of what I found out.

This article is inspired by this question on StackOverflow, which is basically about how Sonarqube and Jacoco calculate coverage.

Line Coverage

The line coverage is the simplest concept. It measures how many executable lines are touched during a test run. As Jacoco operates on byte code and not on source code, it applies some filtering on compiler generated artifacts such as

  • empty constructors
  • catch/finally block generated for try-with-resources

and maybe some others. But apart from that, it’s fairly easy-to-understand metric.

Condition Coverage: The Basics

Lets start with a simple example to explain basic terms.

Given you have some construct as

if(a == 1 && b == 2) {
  //do this
} else {
  //do that
}

You have two branches

  • do this
  • do that

And two conditions

  • a == 1 (cond1)
  • b == 2 (cond2)

If you have two test cases

  • test(a == 1, b == 2)
  • test(a == 2, b == 2)

You’re covering both branches because the combined condition of (cond1 && cond2) is either false or true,

But you only cover cond1 fully and only half of cond2, thats 75% condition coverage.

To get full condition coverage, you need an additional test

  • test(a == 1, b == 1)

What is a branch in Jacoco?

Jacoco calculates the conditions on bytecode level by detecting conditional jumps (like IFNE for IF-NOT-EQUAL).

For a source code statement like

if(a==1 && b==2 && c==3) {
   //branch 1
}  else {
   //branch 2
}

the byte code is conceptually the same as this

if(a==1){ //decision point 1
   //branch 1
  if(b==2) { //decision point 2
     // branch 2
     if (c==3) { //decision point 3
       // branch 3
     } else {
       // branch 4
     }
  } else {
    // branch 5
  }
} else {
  // branch 6
} 

so you will get 6 branches with Jacoco. For the author’s full explanation, see here.

How is Complexity and Branch Coverage calculated in Jacoco?

According to Jacoco documentation, the complexity is calculated by

C = B - D + 1

where

  • B is the number of branches
  • D is the number of decision points.

When you run the CoreTutorial of the Jacoco project, it reports for a class TestTarget

  • 1 of 4 branches missed
  • 1 of 5 complexity missed

For Jacoco, branches and complexity seem to be something different, but also related. How does Jacoco calculate these metrics?

The TestTarget it analyzes looks like this

public static class TestTarget implements Runnable {

    public void run() {
        isPrime(7);
    }

    private boolean isPrime(final int n) {
        for (int i = 2; 
             i * i <= n; // decision point 1 -> branch 1/2 (IFCMPGT)
             i++) {
            if ((n ^ i) == 0) { //decision point 2 -> branch 3/4 (IFNE)
                return false;
            }
        }
        return true;
    }

}

If a method has no decision point it also has no branch and therefore a complexity of 1 (0 - 0 + 1) - makes sense somehow.

The metrics for this class can be broken down to:

  • the (default) constructor: 0 branches, 1 complexity
  • the run method: 0 branches, 1 complexity
  • the isPrime method: 4 branches, 3 complexity (4 - 2 + 1)
  • Jacoco’s Complexity is Branch - DecisionPoints + 1 per Method. Overall complexity is the sum of all methods’ complexities!. Each method contributes at minimum 1 complexity and 0 branch.
  • Jacoco’s Branch Coverage the “branch coverage” in Sonarqube as described in the above theory part.

How does Sonarqube calculate the ‘Coverage’

Line Coverage and Branch Coverage in Sonarqube are used directly from the coverage plugin, i.e. Jacoco

In addition to Line- and Branch Coverage, Sonarqube further calculates a ‘Coverage’ to provide a single metrics for the code coverage. It is a combined metric from the line and branch coverage . It is calculated use this formula

Coverage = (CT + CF + LC)/(2*B + EL)

Where

  • CT = conditions that have been evaluated to ‘true’ at least once
  • CF = conditions that have been evaluated to ‘false’ at least once
  • LC = covered lines = lines_to_cover - uncovered_lines
  • B = total number of conditions
  • EL = total number of executable lines (lines_to_cover)

(see Sonarqube Metric Definition)

How to use the coverage?

The coverage information itself - be it line or branch coverage - itself is a fairly weak indicator for code quality as it can not tell you if the test would fail if the touched code has an actual bug (apart from throwing an unexpected unchecked exception).

A high coverage value does not imply a good test quality. You need other tools - such as Mutation Testing - to determine the effectiveness of your test suite to catch bug.

Nevertheless, a low coverage value is a warning sign that you have no sufficient tests at all. But once you reach a certain value (i.e. 60%) you should complement your unit test approach with mutation testing, which is much more costly but gives you a semantic coverage of your tests/code.