Parallel Test Execution

Basically, there are two ways in maven-failsafe-plugin to achieve parallel test execution.

The most obvious one is by using the parallel parameter. The possible values depend on the test provider used. For JUnit 4.7 and onwards, this may be methods, classes, both, suites, suitesAndClasses, suitesAndMethods, classesAndMethods or all. As a prerequisite in JUnit tests, the JUnit runner should extend org.junit.runners.ParentRunner. If no runner is specified through the annotation @org.junit.runner.RunWith, the prerequisite is accomplished.

As of maven-failsafe-plugin:2.16, the value "both" is deprecated but it still can be used and behaves same as classesAndMethods.

See the example pages for JUnit and TestNG for details.

The extension of the parallelism is configured using the following parameters. The parameter useUnlimitedThreads allows for an unlimited number of threads. Unless useUnlimitedThreads=true, the parameter threadCount can be used with the optional parameter perCoreThreadCount=true (true by default). The parameters useUnlimitedThreads and threadCount are to be interpreted in the context of the value specified for the parallel parameter.

As of maven-failsafe-plugin:2.16, one can impose thread-count limitations on suites, classes or methods using one or more of the parameters threadCountSuites, threadCountClasses and threadCountMethods. If only threadCount is specified, maven-failsafe-plugin attempts to optimize the thread counts for suites, classes and methods and reuses the threads in favor of a leaf, e.g. parallel methods (optionally increasing concurrent methods).

As an example with an unlimited number of threads, there is maximum of three concurrent threads to execute suites: parallel=all, useUnlimitedThreads=true, threadCountSuites=3.

In the second example, the number of concurrent methods is not strictly limited: parallel=classesAndMethods, threadCount=8, threadCountClasses=3. Here the number of parallel methods is varying from 5 to 7. Accordingly parallel=all, but the sum of threadCountSuites and threadCountClasses must not exceed certain (threadCount - 1). Other combinations are possible with unspecified thread-count leaf. Make sure that the leaf is last from the order suites-classes-methods in parallel.

In the third example the thread-counts represent a ratio, e.g. for parallel=all, threadCount=16, threadCountSuites=2, threadCountClasses=3, threadCountMethods=5. Thus the concurrent suites will be 20%, concurrent classes 30%, and concurrent methods 50%.

Finally, the threadCount and useUnlimitedThreads may not be necessarily configured if the equivalent thread-counts are specified for the value in parallel.

The maven-failsafe-plugin is trying to reuse threads, thus optimize the thread-counts, and prefers thread fairness. The optimization parallelOptimized of the number of Threads is enabled by default in terms of e.g. the number of Suite runners do not necessarily have to waste Suite's Thread resources. If threadCount is used, then the leaf with unlimited thread-count may speed up especially at the end of test phase.

The parameters parallelTestsTimeoutInSeconds and parallelTestsTimeoutForcedInSeconds are used to specify an optional timeout in parallel execution. If the timeout is elapsed, the plugin prints the summary log with ERROR lines: "These tests were executed in prior to the shutdown operation", and "These tests are incomplete" if the running Threads were interrupted.

Note: As designed by JUnit runners, the static methods annotated with e.g. @Parameters, @BeforeClass and @AfterClass are called in parent thread. For the sake of memory visibility between threads synchronize the methods. See the keywords: volatile, synchronized, immutable and final in Java Memory Model - JSR-133.

The important thing to remember with the parallel option is: the concurrency happens within the same JVM process. That is efficient in terms of memory and execution time, but you may be more vulnerable towards race conditions or other unexpected and hard to reproduce behavior.

The other possibility for parallel test execution is setting the parameter forkCount to a value higher than 1. The next section covers the details about this and the related reuseForks parameter. Using reuseForks=true (by default) and forking the test classes in reusable JVMs may lead to the same problem with shared static code across @BeforeClass class initializers if using parallel without forking. Therefore setting reuseForks=false may help however it would not guarantee proper functionality of some features, e.g. skipAfterFailureCount.

Parallel Test Execution and Single Thread Execution

As mentioned above the parallel test execution is used with specific thread count. Since of maven-failsafe-plugin:2.18, you can apply the JCIP annotation @net.jcip.annotations.NotThreadSafe on the Java class of JUnit test (pure test class, Suite, Parameterized, etc.) in order to execute it in single Thread instance. The Thread has name maven-surefire-plugin@NotThreadSafe and it is executed at the end of the test run.

Just use project dependency net.jcip:jcip-annotations:1.0 or another artifact com.github.stephenc.jcip:jcip-annotations:1.0-1 with Apache License 2.0.

<dependencies>
  [...]
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <!-- 4.7 or higher -->
    <version>4.7</version>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>com.github.stephenc.jcip</groupId>
    <artifactId>jcip-annotations</artifactId>
    <version>1.0-1</version>
    <scope>test</scope>
  </dependency>
  [...]
</dependencies>

This way the parallel execution of tests classes annotated with @NotThreadSafe are forked in single thread instance (don't mean forked JVM process).

If the Suite or Parameterized is annotated with @NotThreadSafe, the suite classes are executed in single thread. You can also annotate individual test class referenced by Suite, and the other unannotated test classes in the Suite can be subject to run in parallel. This way you can isolate conflicting groups of tests and still run their individual tests in parallel.

Note: As designed by JUnit runners, the static methods annotated with e.g. @Parameters, @BeforeClass and @AfterClass are called in parent thread. Assign classes to @NotThreadSafe Suite to prevent from this trouble. If you do not want to change the hierarchy of your test classes, you may synchronize such methods for the sake of improving memory visibility as a simplistic treatment. See the keywords: volatile, synchronized, immutable and final in Java Memory Model - JSR-133.

Parallel Test-Suite Execution

maven-failsafe-plugin's notion of "suite" is related to junit4 Suite.

For example, say your tests are laid out like this:

src/test/java
+-features.areaA
| +-SomeTest.java
| +-AnotherTest.java
+-features.areaB
| +-DifferentTest.java
| +-OtherTest.java

You would add a TestSuite.java for each package, that would look something like:

package features.areaA;

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({
    SomeTest.class,
    AnotherTest.class
})
public class TestSuite {
}

and similarly for package features.areaB. You would then configure maven-failsafe-plugin as follows:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>3.0.0</version>
  <configuration>
    <includes>
      <include>features.*.TestSuite</include>
    </includes>
    <parallel>suites</parallel>
    <threadCountSuites>2</threadCountSuites>
    <perCoreThreadCount>false</perCoreThreadCount>
  </configuration>
</plugin>

This would then run up to 2 threads, and each thread would be fed a TestSuite.class, one at a time. The test classes within that Suite would be executed sequentially in the order specified the Suite class.

Note that the perCoreThreadCount is a "multiplier" of a sorts. With this parameter set to true (the default value), you will get the specified number of threads per CPU core.

Also note that in this specific case threadCountSuites=2 has the same effect as threadCount=2, since we are using parallel=suites.

Parallel maven-failsafe-plugin Execution in Multi-Module Maven Parallel Build

Maven core allows building modules of multi-module projects in parallel with the command line option -T. This multiplies the extent of concurrency configured directly in maven-failsafe-plugin.

Forked Test Execution

The parameter forkCount defines the maximum number of JVM processes that maven-failsafe-plugin will spawn concurrently to execute the tests. It supports the same syntax as -T in maven-core: if you terminate the value with a 'C', that value will be multiplied with the number of available CPU cores in your system. For example forkCount=2.5C on a Quad-Core system will result in forking up to ten concurrent JVM processes that execute tests.

The parameter reuseForks is used to define whether to terminate the spawned process after one test class and to create a new process for the next test in line (reuseForks=false), or whether to reuse the processes to execute the next tests (reuseForks=true).

The default setting is forkCount=1/reuseForks=true, which means that maven-failsafe-plugin creates one new JVM process to execute all tests in one Maven module.

forkCount=1/reuseForks=false executes each test class in its own JVM process, one after another. It creates the highest level of separation for the test execution, but it would probably also give you the longest execution time of all the available options. Consider it as a last resort.

With the argLine property, you can specify additional parameters to be passed to the forked JVM process, such as memory settings. System property variables from the main maven process are passed to the forked process as well. Additionally, you can use the element systemPropertyVariables to specify variables and values to be added to the system properties during the test execution.

You can use the place holder ${surefire.forkNumber} within argLine, or within the system properties (both those specified via mvn test -D... and via systemPropertyVariables). Before executing the tests, the failsafe plugin replaces that place holder by the number of the actually executing process, counting from 1 to the effective value of forkCount times the maximum number of parallel executions in Maven parallel builds, i.e. the effective value of the -T command line argument of Maven core.

In case of disabled forking (forkCount=0), the place holder will be replaced with 1.

The following is an example configuration that makes use of up to three forked processes that execute the tests and then terminate. A system property databaseSchema is passed to the processes, that shall specify the database schema to use during the tests. The values for that will be MY_TEST_SCHEMA_1, MY_TEST_SCHEMA_2, and MY_TEST_SCHEMA_3 for the three processes. Additionaly by specifying custom workingDirectory each of processes will be executed in a separate working directory to ensure isolation on file system level.

<plugins>
[...]
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0</version>
    <configuration>
        <forkCount>3</forkCount>
        <reuseForks>true</reuseForks>
        <argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
        <systemPropertyVariables>
            <databaseSchema>MY_TEST_SCHEMA_${surefire.forkNumber}</databaseSchema>
        </systemPropertyVariables>
        <workingDirectory>FORK_DIRECTORY_${surefire.forkNumber}</workingDirectory>
    </configuration>
  </plugin>
[...]
</plugins>

In case of a multi module project with tests in different modules, you could also use, say, mvn -T 2 ... to start the build, yielding values for ${surefire.forkNumber} ranging from 1 to 6.

Imagine you execute some tests that use a JPA context, which has a notable initial startup time. By setting reuseForks=true, you can reuse that context for consecutive tests. And as many tests tend to use and access the same test data, you can avoid database locks during the concurrent execution by using distinct but uniform database schemas.

Port numbers and file names are other examples of resources for which it may be hard or undesired to be shared among concurrent test executions.

Isolating report directories across forks

You may run multiple TestNG suites in parallel fork JVM processes. In that case one can see that each process has created XML report file with same name overriding one each other. In order to prevent from this, you may want to change report directory for each Suite XML. Since of Version 2.22.1 you can parameterize reportsDirectory with placeholder ${surefire.forkNumber}.

[...]
<dependencies>
    [...]
    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>5.7</version>
        <classifier>jdk15</classifier>
    </dependency>
    [...]
</dependencies>
[...]
<build>
    <plugins>
            [...]
            <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-failsafe-plugin</artifactId>
                    <version>3.0.0</version>
                    <configuration>
                            <forkCount>2</forkCount>
                            <reuseForks>false</reuseForks>
                            <reportsDirectory>target/surefire-reports-${surefire.forkNumber}</reportsDirectory>
                            <suiteXmlFiles>
                                    <suiteXmlFile>src/test/resources/Suite1.xml</suiteXmlFile>
                                    <suiteXmlFile>src/test/resources/Suite2.xml</suiteXmlFile>
                            </suiteXmlFiles>
                    </configuration>
            </plugin>
        [...]
    </plugins>
</build>

Combining forkCount and parallel

The modes forkCount=0 and forkCount=1/reuseForks=true can be combined freely with the available settings for parallel.

As reuseForks=false creates a new JVM process for each test class, using parallel=classes would have no effect. You can still use parallel=methods, though.

When using reuseForks=true and a forkCount value larger than one, test classes are handed over to the forked process one-by-one. Thus, parallel=classes would not change anything. However, you can use parallel=methods: classes are executed in forkCount concurrent processes, each of the processes can then use threadCount threads to execute the methods of one class in parallel.

Regarding the compatibility with multi-module parallel maven builds via -T, the only limitation is that you can not use it together with forkCount=0.

When running parallel maven builds without forks, all system properties are shared among the builder threads and failsafe executions, therefore the threads will encounter race conditions when setting properties, e.g. baseDir, which may lead to changing system properties and unexpected runtime behaviour.

Known issues and limitations

• ${surefire.forkNumber} propagation is not supported on Maven 2.x (the variable will be resolved to null value all the time)
• ${surefire.forkNumber} is properly propagated within workingDirectory since maven-failsafe-plugin:2.19, more details in SUREFIRE-1136