Fork me on GitHub
Built by Maven

Usage

To use the Failsafe Plugin, you need to add the following configuration to your pom.xml:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.5.1</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

The Failsafe Plugin can be invoked by calling the verify phase of the build lifecycle.

mvn verify

Using Different Testing Providers

Tests in your test source directory can be any combination of the following:

  • TestNG
  • JUnit (3.8, 4.x or 5.x)
  • POJO

Which providers are available is controlled simply by the inclusion of the appropriate dependencies (i.e., junit:junit or junit:junit-dep for JUnit4, junit-jupiter-engine or junit-vintage-engine for JUnit5 and org.testng:testng 4.7+ for TestNG). Since this is required to compile the test classes anyway, no additional configuration is required.

Note that any normal Surefire integration works identically no matter which providers are in use - so you can still produce a Cobertura report and a Surefire results report on your project web site for your TestNG tests, for example.

The POJO provider above allows you to write tests that do not depend on either of JUnit and TestNG. It behaves in the same way, running all test* methods that are public in the class, but the API dependency is not required. To perform assertions, the JDK 1.4 assert keyword can be used. See Using POJO Tests for more information.

All of the providers support the Surefire Plugin parameter configurations. However, there are additional options available if you are running TestNG tests (including if you are using TestNG to execute your JUnit tests, which occurs by default if both are present in Surefire).

See Using TestNG for more information.

Using jetty and maven-failsafe-plugin

You need to bind one of jetty:start, jetty:run, jetty:run-exploded or jetty:run-war to the pre-integration-test phase with daemon set to true, bind failsafe:integration-test to the integration-test phase, bind jetty:stop to the post-integration-test phase and finally bind failsafe:verify to the verify phase. Here is an example:

<project>
  [...]
  <build>
    [...]
    <plugins>
      [...]
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.5.1</version>
        <executions>
          <execution>
            <id>integration-test</id>
            <goals>
              <goal>integration-test</goal>
            </goals>
          </execution>
          <execution>
            <id>verify</id>
            <goals>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <version>9.2.2.v20140723</version>
        [...]
        <configuration>
          [...]
          <scanIntervalSeconds>10</scanIntervalSeconds>
          <stopPort>8005</stopPort>
          <stopKey>STOP</stopKey>
          [...]
        </configuration>
        [...]
        <executions>
          [...]
          <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
              <goal>start</goal>
            </goals>
            <configuration>
              <scanIntervalSeconds>0</scanIntervalSeconds>
              <daemon>true</daemon>
            </configuration>
          </execution>
          <execution>
            <id>stop-jetty</id>
            <phase>post-integration-test</phase>
            <goals>
              <goal>stop</goal>
            </goals>
          </execution>
          [...]
        </executions>
        [...]
      </plugin>
      [...]
    </plugins>
    [...]
  </build>
  [...]
</project>

You then invoke Maven with a phase of verify or later in order to run the integration tests. DO NOT directly invoke any of the phases pre-integration-test, integration-test, or post-integration-test as these are too long to type and they will likely leave a jetty container running.

mvn verify

Note: during test development, you will likely run a jetty instance in the background. to help running the integration tests, it can be handy to bind jetty:stop to the pre-integration-test phase before jetty:run to flush out any running jetty instance before starting the integration test jetty instance, e.g.

<project>
  [...]
  <build>
    [...]
    <plugins>
      [...]
      <plugin>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <version>9.2.2.v20140723</version>
        [...]
        <executions>
          [...]
          <execution>
            <id>start-jetty</id>
            <phase>pre-integration-test</phase>
            <goals>
              <!-- stop any previous instance to free up the port -->
              <goal>stop</goal>
              <goal>start</goal>
            </goals>
            [...]
          </execution>
          [...]
        </executions>
        [...]
      </plugin>
      [...]
    </plugins>
    [...]
  </build>
  [...]
</project>

Reporting integration test results

The Failsafe Plugin uses the exact same format as the Surefire Plugin, so to generate a report you just add a second Surefire Report Plugin report set using the Failsafe reports directory, e.g.

<project>
  [...]
  <reporting>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-report-plugin</artifactId>
        <version>3.5.1</version>
        <reportSets>
          <reportSet>
            <id>integration-tests</id>
            <reports>
              <report>failsafe-report-only</report>
            </reports>
          </reportSet>
        </reportSets>
      </plugin>
    </plugins>
  </reporting>
  [...]
</project>

Running integration tests multiple times

If you need to run your integration tests multiple times, just use multiple executions of the integration-test goal. You will need to specify a different summary file for each execution, and then configure the verify goal with the multiple summary files in order to fail the build when any one execution has failures, e.g.

<project>
  [...]
  <reporting>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.5.1</version>
        <executions>
          <execution>
            <id>integration-test-red-bevels</id>
            <goals>
              <goal>integration-test</goal>
            </goals>
            <configuration>
              <systemPropertyVariables>
                <bevels>red</bevels>
              </systemPropertyVariables>
              <summaryFile>target/failsafe-reports/failsafe-summary-red-bevels.xml</summaryFile>
            </configuration>
          </execution>
          <execution>
            <id>integration-test-no-bevels</id>
            <goals>
              <goal>integration-test</goal>
            </goals>
            <configuration>
              <systemPropertyVariables>
                <bevels>none</bevels>
              </systemPropertyVariables>
              <summaryFile>target/failsafe-reports/failsafe-summary-no-bevels.xml</summaryFile>
            </configuration>
          </execution>
          <execution>
            <id>verify</id>
            <goals>
              <goal>verify</goal>
            </goals>
            <configuration>
              <summaryFiles>
                <summaryFile>target/failsafe-reports/failsafe-summary-red-bevels.xml</summaryFile>
                <summaryFile>target/failsafe-reports/failsafe-summary-no-bevels.xml</summaryFile>
              </summaryFiles>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </reporting>
  [...]
</project>

Usage in multi-module projects

The recommendations for using the Failsafe Plugin listed at the top of this page are fine for 95% of use cases. When you are defining a shared definition of the Failsafe Plugin in a parent pom, it is considered best practice to define an execution id in order to allow child projects to override the configuration. Thus you might have the following in your parent pom.xml:

<project>
  [...]
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-failsafe-plugin</artifactId>
          <version>3.5.1</version>
          <executions>
            <execution>
              <id>integration-test</id>
              <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
  [...]
</project>

The child projects can then trigger usage of the Failsafe Plugin with

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.5.1</version>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

For very complex builds, it may be better to separate the executions for the integration-test and verify goals.

<project>
  [...]
  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-failsafe-plugin</artifactId>
          <version>3.5.1</version>
          <executions>
            <execution>
              <id>integration-test</id>
              <goals>
                <goal>integration-test</goal>
              </goals>
            </execution>
            <execution>
              <id>verify</id>
              <goals>
                <goal>verify</goal>
              </goals>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
  [...]
</project>