Parallel execution of Behat tests

In this article i want to describe how execute Behat tests in parallel from local machine (Windows) or Bamboo server (Red Hat). 

Main questions:
1. Parallel execution on Windows
2. Parallel execution from Bamboo 

3. Consolidated report 

1. Parallel execution on Windows

On local machine i use GNU Parallel which can be installed under Microsoft Windows, details here.
You should setup Cygwin and install here "make" and "perl" packages. Or download ready for use Cygwin with all packages here.
Also you need Selenium Grid.


For parallel execution Behat tests in Cygwin terminal use this  script:

find features/SomeSuiteFolder -iname "*.feature" | parallel -j4 '../vendor/behat/behat/bin/behat --format=html --tags  @ featureOrTestTag --config behat.yml' {} 

This script find all files with name "*.feature" and execute all found features with tags @featureOrTestTag in parallel
-j4 - tread count.

This solution use parallelisation on OS level. It means that each thread will be executed on one processor core. So there is some limitation for threads count. For example if you have processor with 4 cores you can execute in parallel only 4 tests.  But if you have multi processor server you can get as much parallel threads as cores available.

2. Parallel execution from Bamboo 

Now talk about CI and Bamboo.  You can also execute your Behat tests in parallel with GNU Parallel but you should install it before. 

For parallel execution on CI we use xargs because of it has the advantage of being installed by default on most systems.

Script for execution:
cd ./Tests
grep "@featureOrTestTag" -lR --null features/SomeSuiteFolder/ | xargs --null -- parallel -j8 -i ../vendor/behat/behat/bin/behat --format="html"  --suite=suiteName --tags '@featureOrTestTag' --config behat.yml {} -- &

This script also find all features with tags "@featureOrTestTag" and execute all found features in parallel. As we have multi processor server we can execute in parallel up to 32 threads.

Be sure that tags in grep "@featureOrTestTag"  and  --tags '@featureOrTestTag' are the same.
Also be sure that you didn't have features or tests with tags like @featureOrTestTagSomeText because of grep will find all features @featureOrTestTag and @featureOrTestTagSomeText and when parallel will execute recursively behat will be not able to run features and tests with  @featureOrTestTagSomeText tag and you get error and failed build.

3. Consolidated report

Of course parallelisation without test report not very useful. Feature of this approach is that each test is running as independent behat process and at the end we want to get consolidated report of all features. It was implemented with Behat report plugin BehatHtmlFormatterPlugin.  I refactor some code and optimise it for parallel execution (see forked version of BehatHtmlFormatterPlugin).

How does it work?
When you execute you tests in parallel for each feature will be created report in folder /build/html/behat/ and one index.html with list of executed features in left side with statistics and status. During navigation on executed features from left menu reports will be loaded in right part of the window in iframe.
Report example:

Enjoy with parallelisation and don't hesitate to ask me questions