Choosing the right forking strategy and parallel execution settings can have substantial impact on the memory requirements and the execution time of your build system.
Surefire offers a variety of options to execute tests in parallel, allowing you to make best use of the hardware at your disposal. But forking in particular can also help keeping the memory requirements low.
This page shall give you some ideas of how you can configure the test execution in a way best suitable for your environment.
Basically, there are two ways in Surefire to achieve parallel test execution.
The most obvious one is by using the parallel parameter. The possible values depend on the test provider used. For JUnit 4.7 and onwards, this may be methods, classes, both, suites, suitesAndClasses, suitesAndMethods, classesAndMethods or all. As a prerequisite in JUnit tests, the JUnit runner should extend org.junit.runners.ParentRunner. If no runner is specified through the annotation @org.junit.runner.RunWith, the prerequisite is accomplished.
As of Surefire 2.16, the value "both" is deprecated but it still can be used and behaves same as classesAndMethods.
See the example pages for JUnit and TestNG for details.
The extent of the parallelism is configured using the following parameters. The parameter useUnlimitedThreads allows for an unlimited number of threads. Unless useUnlimitedThreads is set to "true", the parameter threadCount can be used with the optional parameter perCoreThreadCount. The parameters useUnlimitedThreads and threadCount are to be interpreted in the context of the value specified for the parallel parameter.
One can impose thread-count limitations on suites, classes or methods using one or more of the parameters threadCountSuites, threadCountClasses and threadCountMethods. If only threadCount is specified, Surefire attempts to estimate the thread counts for suites, classes and methods and reuses the threads in favor of a leaf, e.g. parallel methods (possibly increasing concurrent methods).
As an example with an unlimited number of threads, there is maximum of three concurrent threads to execute suites: parallel = all, useUnlimitedThreads = true, threadCountSuites = 3.
In the second example, the number of concurrent methods is not strictly limited: parallel = classesAndMethods, threadCount = 8, threadCountClasses = 3. Here the number of parallel methods is varying from 5 to 7. Accordingly parallel = all, but the sum of threadCountSuites and threadCountClasses must not exceed certain (threadCount - 1). Other combinations are possible with unspecified thread-count leaf. Make sure that the leaf is last from the order suites-classes-methods in parallel.
In the third example the thread-counts represent a ratio, e.g. for parallel = all, threadCount = 16, threadCountSuites = 2, threadCountClasses = 3, threadCountMethods = 5. Thus the concurrent suites will be 20%, concurrent classes 30%, and concurrent methods 50%.
Finally, the threadCount and useUnlimitedThreads may not be necessarily configured if the equivalent thread-counts are specified for the value in parallel.
The surefire is always trying to reuse threads, optimize the thread-counts, and prefers thread fairness.
The parameters parallelTestsTimeoutInSeconds and parallelTestsTimeoutForcedInSeconds are used to specify an optional timeout in parallel execution. If the timeout is elapsed, the plugin prints the summary log with ERROR lines: "These tests were executed in prior to the shutdown operation", and "These tests are incomplete" if the running Threads were interrupted.
The important thing to remember with the parallel option is: the concurrency happens within the same JVM process. That is efficient in terms of memory and execution time, but you may be more vulnerable towards race conditions or other unexpected and hard to reproduce behavior.
The other possibility for parallel test execution is setting the parameter forkCount to a value higher than 1. The next section covers the details about this and the related reuseForks property.
As mentioned above the parallel test execution is used with specific thread count. Since of Surefire 2.18, you can apply the JCIP annotation @net.jcip.annotations.NotThreadSafe on the Java class of JUnit test (test class, Suite, Parameterized, etc.) in order to execute it in single Thread instance. The Thread has name "maven-surefire-plugin@NotThreadSafe".
Just use the dependency net.jcip:jcip-annotations:1.0, or another Artifact with Apache License com.github.stephenc.jcip:jcip-annotations:1.0-1.
This way the parallel execution of tests classes annotated with @NotThreadSafe are forked in single thread instance (don't mean forked JVM process).
If the Suite or Parameterized is annotated with @NotThreadSafe, the suite classes are executed in single thread. You can also annotate individual test class referenced by Suite, and the other unannotated test classes in the Suite can be subject to run in parallel.
Note: As designed by JUnit runners, the static methods annotated with @BeforeClass and @AfterClass are called in parent thread. Assign classes to the @NotThreadSafe Suite to prevent from this trouble.
Maven core allows building modules of multi-module projects in parallel with the command line option -T. This multiplies the extent of concurrency configured directly in Surefire.
The parameter forkCount defines the maximum number of JVM processes that Surefire will spawn concurrently to execute the tests. It supports the same syntax as -T in maven-core: if you terminate the value with a 'C', that value will be multiplied with the number of available CPU cores in your system. For example forkCount=2.5C on a Quad-Core system will result in forking up to ten concurrent JVM processes that execute tests.
The parameter reuseForks is used to define whether to terminate the spawned process after one test class and to create a new process for the next test in line (reuseForks=false), or whether to reuse the processes to execute the next tests (reuseForks=true).
The default setting is forkCount=1/reuseForks=true, which means that Surefire creates one new JVM process to execute all tests in one maven module.
forkCount=1/reuseForks=false executes each test class in its own JVM process, one after another. It creates the highest level of separation for the test execution, but it would probably also give you the longest execution time of all the available options. Consider it as a last resort.
With the argLine property, you can specify additional parameters to be passed to the forked JVM process, such as memory settings. System property variables from the main maven process are passed to the forked process as well. Additionally, you can use the element systemPropertyVariables to specify variables and values to be added to the system properties during the test execution.
You can use the place holder ${surefire.forkNumber} within argLine, or within the system properties (both those specified via mvn test -D... and via systemPropertyVariables). Before executing the tests, Surefire replaces that place holder by the number of the actually executing process, counting from 1 to the effective value of forkCount times the maximum number of parallel Surefire executions in maven parallel builds, i.e. the effective value of the -T command line argument of maven core.
In case of disabled forking (forkCount=0), the place holder will be replaced with 1.
The following is an example configuration that makes use of up to three forked processes that execute the tests and then terminate. A system property databaseSchema is passed to the processes, that shall specify the database schema to use during the tests. The values for that will be MY_TEST_SCHEMA_1, MY_TEST_SCHEMA_2, and MY_TEST_SCHEMA_3 for the three processes.
<plugins>
[...]
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.18.1</version>
<configuration>
<forkCount>3</forkCount>
<reuseForks>true</reuseForks>
<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
<systemPropertyVariables>
<databaseSchema>MY_TEST_SCHEMA_${surefire.forkNumber}</databaseSchema>
</systemPropertyVariables>
</configuration>
</plugin>
[...]
</plugins>
In case of a multi module project with tests in different modules, you could also use, say, mvn -T 2 ... to start the build, yielding values for ${surefire.forkNumber} ranging from 1 to 6.
Imagine you execute some tests that use a JPA context, which has a notable initial startup time. By setting reuseForks=true, you can reuse that context for consecutive tests. And as many tests tend to use and access the same test data, you can avoid database locks during the concurrent execution by using distinct but uniform database schemas.
Port numbers and file names are other examples of resources for which it may be hard or undesired to be shared among concurrent test executions.
The modes forkCount=0 and forkCount=1/reuseForks=true can be combined freely with the available settings for parallel.
As reuseForks=false creates a new JVM process for each test class, using parallel=classes would have no effect. You can still use parallel=methods, though.
When using reuseForks=true and a forkCount value larger than one, test classes are handed over to the forked process one-by-one. Thus, parallel=classes would not change anything. However, you can use parallel=methods: classes are executed in forkCount concurrent processes, each of the processes can then use threadCount threads to execute the methods of one class in parallel.
Regarding the compatibility with multi-module parallel maven builds via -T, the only limitation is that you can not use it together with forkCount=0.
Surefire versions prior 2.14 used the parameter forkMode to configure forking. Although that parameter is still supported for backward compatibility, users are strongly encouraged to migrate their configuration and use forkCount and reuseForks instead.
The migration is quite simple, given the following mapping:
| Old Setting | New Setting |
| forkMode=once (default) | forkCount=1 (default), reuseForks=true (default) |
| forkMode=always | forkCount=1 (default), reuseForks=false |
| forkMode=never | forkCount=0 |
| forkMode=perthread, threadCount=N | forkCount=N, (reuseForks=false, if you did not had that one set) |