The Holy Java

Building the right thing, building it right, fast

Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo

Posted by Jakub Holý on February 5, 2012

Goal: Execute the slow integration tests separately from unit tests and show as much information about them as possible in Sonar.

The first part – executing IT and UT separately – is achieved by using the maven-failsafe-plugin and by naming the integration tests *IT (so that the unit test running surefire-maven-plugin will ignore them while failsafe will execute them in the integration-test phase and collect results in the verify phase).

The second part – showing information about integration tests in Sonar – is little more tricky. Metrics of integration tests will not be included in the Test coverage + Unit tests success widget. You can add Integration test coverage (IT coverage) widget if you enable JaCoCo but there is no alternative for the test success metrics. But don’t despair, read on!

Important notice: The integration of Sonar, JaCoCo and Failsafe evolves quite quickly so this information may easily get outdated with the next releases of Sonar

Versions: Sonar 2.12, Maven 3.0.3

Prerequisity: How Sonar Works

It’s important to understand how Sonar works and how it is integrated into the build process. Briefly:

  • Sonar is run after the project is built (and thus various artifacts are already generated): either as a post- action in Jenkins or after executing mvn install manually
  • Sonar comes bundled with various integrated quality analysis plugins such as PMD, Checkstyle, Findbugs (depending on the quality profile chosen). You do not need to add them manually to your POM (but you can, if you need to configure them). However if you need something it doesn’t do yet such as collecting coverage for integration tests, you have to do it manually.
  • Sonar may either reuse existing reports (checkstyle, …) or generate its own.
  • F.ex. if you choose JaCoCo as the default code coverage provider, Sonar will automatically rerun unit tests with JaCoCo enabled to collect the coverage metrics. You can see the file target/sonar/sonar-pom.xml that it generates to see what it does.

Fooling Sonar to Display IT Test Success

Executing unit tests via Surefire and integration tests via Failsafe results in not beeing able to see when integration tests fail. (Because this is shown by the Unit test success widget, which doesn’t support Failsafe.) However it’s possible to fool Sonar to show test success for both unit and integration tests together by instructing Failsafe to store its test reports to the same directory as Surefire instead of the default failsafe-reports, and that’s what we do:

<!-- pom.xml, the build/plugins section -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <configuration>
        <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
    </configuration>
</plugin>

How to display IT Code Coverage in Sonar

Sonar 2.12 currently isn’t able to compute test coverage for integration tests automatically and thus we have to do it manually. Summary:

  1. Add Failsafe to your Maven configuration as shown above.
  2. Add JaCoCo to Maven and combine it with Failsafe to produce the coverage report when integration tests are run
  3. Instruct Sonar to execute the verify phase and pass the path to the integration test report to it

1. Add Failsafe to your Maven configuration as shown above

See the pom.xml fragment under “Fooling Sonar to Display IT Test Success” above.

2. Add JaCoCo to Maven and combine it with Failsafe to produce the coverage report when integration tests are run

As Sonar 2.12 doesn’t automatically reconfigure Failsafe to collect code coverage, we have to instruct Failsafe manually to load the JaCoCo java agent that will collect and store the coverage report (to target/jacoco.exec by default):

<!-- pom.xml fragment: -->
<build>
 ...
     <!-- Run integration tests (*IT) -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <configuration>
            <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
            <argLine>${jacoco.agent.argLine}</argLine>
        </configuration>
    </plugin>
    <!--
        Compute integration test coverage for Sonar
        BEWARE: Sonar doesn't run the verify phase, it has to be forced by setting -Dsonar.phase=verify
    -->
    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <configuration>
            <propertyName>jacoco.agent.argLine</propertyName> <!-- default: argLine -->
            <includes>
                <include>com/comoyo/**</include>
            </includes>
            <destFile>${project.build.directory}/jacoco-integration.exec</destFile> <!-- agent -->
            <dataFile>${project.build.directory}/jacoco-integration.exec</dataFile> <!-- report -->
        </configuration>
        <executions>
            <execution>
                <id>agent</id>
                <goals><goal>prepare-agent</goal></goals>
            </execution>
        </executions>
    </plugin>
</build>

The key lines are 10 (argLine) and 28 (prepare-agent before verify – the default phase, if unspecified, is init).

3. Instruct Sonar to execute the verify phase and pass the path to the integration test report to it

Important: If you use Sonar prior to 2.12 then you must install the Sonar JaCoCo plugin manually.

Whether running Sonar from Jenkins or locally we have to make sure that the verify phase is run and pass to Sonar the path to the generated JaCoCo integration test coverage report. This is best done in Sonar in the JaCoCo plugin configuration (on the project level?) but you could also do it manually by passing it to maven via “-Dsonar.jacoco.itReportPath=target/jacoco-integration.exec” or by setting the property in pom.xml under project/properties.

Jenkins Sonar plugin does not require any special configuration if path to the report is set inside Sonar’s JaCoCo plugin.

Command line (the recommended way; the first line runs among others the phase verify and thus generates the report):

payment$ mvn clean install -Dtest=false -DfailIfNoTests=false
payment$ mvn sonar:sonar

We could actually invoke Maven with a single line, though that isn’t recommended because it would run tests multiple times:

payment$ mvn -Dsonar.phase=verify sonar:sonar
NOTE: If you set JaCoCo as the default code coverage provider in Sonar than it will produce coverage for unit tests overriding the integration test coverage from the verify phase. Solutions: 1) Don’t enable JaCoCo, 2) Configure JaCoCo to use a different file for integration tests (which we do)

Some logs of interest from running mvn sonar:sonar:

...
JaCoCo agent (version 0.5.3.201107060350) extracted: /var/folders/k0/2842tm752zv1dh4q77_gmgdr0000gn/T/jacocoagent2548420105762793132.jar
JVM options: -javaagent:/var/folders/k0/2842tm752zv1dh4q77_gmgdr0000gn/T/jacocoagent2548420105762793132.jar=destfile=target/jacoco.exec,excludes=*_javassist_*
Initializer FindbugsMavenInitializer...
Initializer FindbugsMavenInitializer done: 4 ms
Execute maven plugin maven-surefire-plugin...
Execute org.apache.maven.plugins:maven-surefire-plugin:2.8.1:test...
...
Execute maven plugin maven-surefire-plugin done: 9856 ms
Initializer JacocoMavenInitializer...
..
Sensor SquidSensor done: 2207 ms
Sensor JaCoCoSensor...
Analysing /myproject/target/jacoco.exec
Sensor JaCoCoSensor done: 559 ms
Sensor JaCoCoItSensor...
Analysing /myproject/target/jacoco-integration.exec
Sensor JaCoCoItSensor done: 89 ms
Sensor SurefireSensor...
  • Notice that I’ve JaCoCo set as my code coverage provider in Sonar and Sonar does use its own copy of it  (the line JaCoCo agent .. extracted), which it uses in the test phase
  • Notice that Sonar runs surefire:test automatically (with instrumentation) to collect code coverage
  • Notice that JaCoCo processed both *.exec files (the first generated by Sonar for unit tests, the other generated by Maven in verify prior to calling Sonar)

Tip: Compute the total code coverage of unit + integration tests

Unit and integration test coverage are computed separately; to see the total code coverage we would need to merge the two (notice we can’t just sum them as both kinds of tests can cover some of the same lines). It would be possible by using JaCoCo both to compute unit test coverage (supported out of the box) and integration test coverage into different files and using its Ant task to merge the two coverage files, passing the resulting file as the IT test coverage file to Sonar (for we cannot get a 3rd widget to display this summed coverage). However I haven’t tried it.

Note on Testing JaCoCo, Failsafe and Maven Integration

If Sonar doesn’t show the IT coverage widget though it is on the dashboard (click on Configure widgets while logged in) or shows 0% though it should be higher, you may check the data that JaCoCo is generating by producing HTML report from them. There is both an Ant task for that, which didn’t work for me, and Maven goal. This is how you instruct JaCoCo to generate the report when “mvn site” is run – notice the lines 13, 14 (remember that you must run “mvn verify” first to generate the binary coverage report):

<!-- pom.xml build/plugins fragment -->
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>agent</id>
            <phase>pre-integration-test</phase>
            <goals><goal>prepare-agent</goal></goals>
        </execution>
        <execution>
            <id>report</id>
            <phase>site</phase>
            <goals><goal>report</goal></goals>
        </execution>
    </executions>
</plugin>

References

JaCoCo Maven plugin has minimal documentation but you can get useful info via:

mvn help:describe -Dplugin=org.jacoco:jacoco-maven-plugin -Ddetail

Sonar: Measure Code Coverage by Integration Tests with Sonar (9/2010) – doesn’t use the JaCoCo Maven plugin and thus has to configure the argLine manually

The Complete POM

See the complete POM at GitHub.

About these ads

5 Responses to “Separating Integration and Unit Tests with Maven, Sonar, Failsafe, and JaCoCo”

  1. Hi,

    I am trying to follow your article but my unit tests run twice when I run mvn /mvn -Dsonar.phase=verify sonar:sonar -Pjbossas-remote-6. The difference from your pom is that in my profile I add some other test resources and libraries my tests need. And if I run mvn clean install -Dtest=false -DfailIfNoTests=false and then sonar:sonar my integration tests coverage data doesnt seem to be generated

    • Hi Kelly,
      – which tests are run twice, unit or integration?
      – who runs them (for 1st/2nd time), Sonar?
      – have you configured Sonar to reuse existing reports or run its own tests?

      • - which tests are run twice, unit or integration?
        unit

        – who runs them (for 1st/2nd time), Sonar?
        1st time seems to be surefire plugin and the 2nd sonar

        – have you configured Sonar to reuse existing reports or run its own tests?
        run its own

      • Then it’s expected that the tests are run twice. Sonar needs to inject the coverage collection calls into the code and run the modified tests again to collect coverage. But there may be also other reasons, try googling “sonar executing tests twice”.

  2. […] (typeof(addthis_share) == "undefined"){ addthis_share = [];}This blog post explains how to separate integration and unit tests with Maven, Sonar, Failsafe and JaCoCo. This is achieved by executing unit tests via Surefire and integration tests via Failsafe. Then […]

Sorry, the comment form is closed at this time.

 
%d bloggers like this: