Parallel CI – Comparing CI Systems Parallelization

In my last blog post on the subject, I wrote about parallelization solutions that are independent of your CI/CD platform. In this follow-up post about Parallel CI, I will review the available parallelization features of common CI/CD platforms, some of which we reviewed in previous blog posts: Jenkins, Bamboo, GitLab, CircleCI, and Bitbucket. These CI systems allow you to parallelize beyond the limitations of a single machine and with simpler declarative syntax and Web UI for status reporting and build logs. 

Why Do We Want to Parallelize Beyond a Single Machine?

As your project grows and becomes more complex (more on code inflation here), so do the requirements from the build process. Running integration tests may require setting up supporting infrastructure and performing other time-consuming tasks. While a single machine nowadays can be quite powerful, eventually you may reach some limits. Additionally, you may want to run some processes on different operating systems or on machines with different hardware (for example – GPU for machine learning). For these reasons the requirement for parallelizing beyond a single machine is quite common.  

Most CI/CD platforms provide a perfect solution for these requirements. Whether it is a self-hosted solution like Jenkins, which can integrate with other solutions to set up additional machines (for example – using Kubernetes or AWS). Or, using a CI/CD SAAS solution that provides the infrastructure you need. 

Jenkins Pipeline – Running Concurrent Jobs

Jenkins supports many different ways of defining jobs and with plugins, even more ways are possible. I will focus on the latest and recommended method of using the declarative pipeline with no additional plugins.  

Let’s start with the simpler parallel directive which can appear under any stage in the pipeline (except if that stage is already under a parallel directive – nested parallel directives are not supported). Under the parallel directive, you may define stages and steps using all available standard directives, including nested stages (as long as they are not parallel themselves).  

Within the parallel stages, you can specify the agent label forcing tests to run on different servers as required by each stage.  

The following example assumes a Makefile with default task which compiles to a binary, with additional tasks for publishing to / downloading from artifact storage and running tests on the binary: 

pipeline { 
    agent any 
    stages { 
        stage('Compile & Publish Binary') { 
            steps { 
                make && make publish 
            } 
        } 
        stage('Run Tests') { 
            when { 
                branch 'master' 
            } 
            failFast true 
            parallel { 
                stage('Test A') { 
                    agent { 
                        label "for-test-a" 
                    } 
                    steps { 
                        make download && make test_a 
                    } 
                } 
                stage('Test B') { 
                    agent { 
                        label "for-test-b"
                    } 
                    steps { 
                        make download && make test_b 
                    } 
                } 
            } 
        } 
    } 
} 

In the previous example, the parallel test stages will start only after the compile & publish stage completes successfully. We also included the “failFast true” directive, so the build will fail immediately when one of the tests fails and any further stages will not run. If we had not included the “failFast true” directive, then the Jenkins runner will wait for both test stages to complete (whether successful or not) and only then the build will end and the next stage will start.  

The matrix directive is a bit more complex but is very useful in many cases, especially for testing on different platforms. Under the matrix directive, you define a list of axes, each axis containing a name and list of values. All the different combinations of axis values are combined and all run in parallel. Many directives like the agent directive can be set under the matrix directive and use the expanded axis values.  

In the following example we extend the previous example to run the tests on different operating systems: 

pipeline { 
    agent any 
    stages { 
        stage('Compile & Publish Binary') { 
            steps { 
                make && make publish 
            } 
        } 
        stage('Run Tests') { 
            when { 
                branch 'master' 
            } 
            failFast true 
            matrix { 
                agent { 
                    label "${PLATFORM}-for-test-${TEST}" 
                } 
    axes { 
        axis { 
            name 'PLATFORM' 
            values 'linux', 'mac', 'windows' 
        } 
        axis { 
            name 'TEST' 
            values 'a','b'
        } 
    } 
    stages { 
                    stage('Test') { 
                        steps { 
                            make download && make test_${TEST} 
                        } 
                    } 
                } 
            } 
        } 
    } 
} 

In this example, the tests will run 6 times for each of the combinations of PLATFORM and TEST. We use the agent directive to select a different server for each combination and we use the TEST variable within the executed stage.  

While the matrix feature is useful for naturally parallelized tasks like multi-platform testing, many other tasks are much harder to split. A compilation process of a C++ application, which produces a single binary, cannot be easily split. This is where solutions like Incredibuild with Jenkins integration shine and allow you to optimize beyond the inherent limits of your build process. 

Related: Advanced Jenkins Parallel Builds (And Jenkins Distributed Builds)

CircleCI- Running Parallel Steps

CircleCI pipelines workflows feature provides a relatively simple way to configure a collection of jobs and their run order. This can be combined with the job parallelism attribute that launches a number of executors for a job, with each executor handling part of the processing for that job.  

Using the workflows feature, you first define the jobs you want to run and the steps to run within each job, you then configure workflows, which are a nested list of jobs which all run in parallel. You can optionally include dependencies between jobs, allowing you to wait for a certain job to complete before starting other jobs. 

First, we will define the jobs, these are the basic building blocks which we use in the workflows: 

jobs: 
  compile_and_publish: 
    docker: 
      image: circleci/buildpack-deps:focal 
    steps: 
      - checkout 
      - run: 
          command: make && make publish 
  make: 
    parameters: 
      task: 
        type: string 
        default: "" 
    docker: 
      image: circleci/buildpack-deps:focal 
    steps:
      - checkout 
      - run: 
          command: make << parameters.task >>

Then we can define the workflows combining these jobs in certain execution order: 

workflows: 
  version: 2 
  compile_publish_and_test: 
    jobs: 
      - compile_and_publish 
      - make: 
          task: test_a 
          requires: compile_and_publish 
      - make: 
          task: test_b 
          requires: compile_and_publish 

In the above example, the compile_and_publish job will run first, and only if it succeeded, the test jobs will then run in parallel. CircleCI will wait for both the test jobs to complete (whether successfully or not) before continuing to the next job. However, job status is reported in real-time and using the CircleCI UI you can cancel the run or retry a specific job without waiting for the entire workflow to complete. CircleCI has a lot more options to configure job dependencies, check out the documentation for more details: The when attribute, Background commands, Ending a job from within a step, The when step, requires, using when in workflows. 

We can further extend workflows using matrix jobs, this allows us to run a job using a combination of given parameters: 

workflows: 
  version: 2 
  compile_publish_and_test: 
    jobs: 
      - compile_and_publish 
      - make: 
          requires: compile_and_publish 
          matrix: 
            parameters: 
              test: [test_a, test_b] 

If our make task had an additional platform parameter, we could run the following to execute all combinations of the given parameter values: 

matrix: 
            parameters: 
              test: [test_a, test_b] 
              platform: [windows, mac, linux] 

CircleCI has another very useful feature to support parallelization within a single job. Let’s say you have a single job which runs make tasks defined in a text file, one make task per line: 

jobs: 
  tests: 
    docker: 
      image: circleci/buildpack-deps:focal 
    steps: 
      - checkout 
      - run: 
          command: while read TASK; do make $TASK; done < tests.txt 

Using the CircleCI parallelism attribute you specify how many executors you want to start, and then you use the CircleCI CLI tool to run the same task on multiple executors. This allows you to implement the task in such a way that each executor will run some of the tasks. Note that this option does not allow splitting an existing make task, as you are still limited by your build system, but it does help to split a group of tasks automatically and run each independent task in parallel: 

jobs: 
  tests: 
    docker: 
      image: circleci/buildpack-deps:focal 
    parallelism: 4    
    steps: 
      - checkout 
      - run: 
          command: for TASK in $(circleci tests split tests.txt); do make $TASK; done 

The above example will launch 4 executors and each executor will run some of the tasks which are defined in tests.txt file. The CircleCI tests split command has a few different options, including a very useful option to split based on timing data, see the CircleCI parallelism documentation for more details. 

CircleCI has many useful parallelization features and is the most feature-rich from the reviewed CI systems. The option to split based on timing data is especially useful but quite complex so I encourage you to learn about in more details from the documentation. As mentioned above, these useful features still do not help you solve the inherent limitations of your build system and allow you to split up a single monolithic task, like C++ compilation, to parallelize even further (and again, for large C++ projects this extra optimization may be critical and this is where Incredibuild’s solution comes naturally in). 

Atlassian Bamboo 

There are several methods to define CI/CD pipelines in Bamboo, we will focus on the Bamboo specs yaml, but the same concepts can be applied to the other methods as well.  

Bamboo supports only basic parallelization, running all jobs under a defined stage in parallel, so nothing special is needed to support parallelization: 

version: 2 
plan: 
  project-key: MYPROJECT 
  name: My Project 
  key: BUILD 
 

stages: 
  - Compile and publish: 
      jobs:  
        - Compile and publish 
  - Test: 
      jobs: 
        - Test A 
        - Test B 
 

Compile and publish: 
  tasks: 
    - script: 
        - make && make publish 
 

Test A: 
  tasks: 
    - script:  
        - make test_a 
 

Test B: 
  tasks: 
    - script:  
        - make test_b

In this example, test a and test b will run in parallel only after the compile and publish stage completes successfully. Bamboo doesn’t have any directives to control dependencies between tasks, so both parallel tasks will run to completion. There is the concept of Final tasks which will always be executed, regardless of whether any build tasks or other final tasks fail. Final tasks will be executed even if you stop the build manually. Read more about this concept in the documentation.  

Unfortunately, Bamboo has quite limited parallelization options compared to the other CI systems, so you will have to rely on your build system or DevOps engineers to further optimize your build process. (And again, for C++ builds, Incredibuild’s solution may come in to help in achieving the required optimization and allowing your developers to focus on development instead of waiting for the build to complete). 

Bitbucket Pipelines- Parallel Steps

Bitbucket Pipelines supports very simple and intuitive parallel steps using the parallel directive: 

pipelines: 
  default: 
    - step: 
        script: 
          - make && make publish 
    - parallel: 
        - step: 
            script: 
              - make test_a 
        - step: 
            script: 
              - make test_b 

In this example, same as the previous examples, the tests are run in parallel only after the build step completes successfully. Both test steps will run in parallel to completion regardless of whether they succeed or fail. BitBucket doesn’t have options to configure dependencies between tasks. See the full documentation for parallel steps for more details.  

The Bitbucket parallelization features are very limited and quite similar to Bamboo (and in fact, both solutions are owned by Atlassian). It means that most of the parallelization and build optimization work will have to be done by your DevOps engineers (and again, for C++ builds we of course recommend getting the help of Incredibuild’s solutions). 

Parallelization With GitLab CI 

GitLab supports parallelization without any specific directive. All the jobs in the same stage run in parallel, this works the same way for any pipeline architecture: 

stages: 
  - build 
  - test 
 

image: ubuntu 
 

build_and_publish: 
  stage: build 
  script: 
    - make && make publish 
 

test_a: 
  stage: test 
  script: 
    - make test_a 
 

test_b: 
  stage: test 
  script: 
    - make test_b 

In the above example, test_a and test_b belong to the same stage, so they will run in parallel only after the build_and_publish stage is completed successfully. GitLab has highly advanced configuration options for dependencies between tasks and stages, read the relevant documentation for more details: directed acyclic graph, child/parent pipelines  

GitLab also supports parallelization within a single job. The parallel attribute configures how many instances of the job to start. Each job instance will receive environment variables containing the total number of instances started and the index of each job instance: 

test: 
  stage: test 
  parallel: 10 
  script: 
    - echo Running job instance $CI_NODE_INDEX out of a total of $CI_NODE_TOTAL jobs 

The above example will start 10 job instances in parallel, each job instance run will show a different job index in the CI_NODE_INDEX environment variables with CI_NODE_TOTAL environment variable value being 10 – the total number of jobs.  

Another option available using the parallel attribute of a job is to define a matrix of values. Using this feature you define a matrix of values and parallel job instances start to handle all combination of the values: 

test: 
  stage: test 
  parallel: 
    matrix: 
      - PLATFORM: windows 
        OS: ["8", "8.1", "10"] 
      - PLATFORM: linux 
        OS: [ubuntu, debian, fedora] 
      - PLATFORM: mac 
        OS: [mojave, catalina, bigsur] 
  script:  
    - echo test $CI_NODE_INDEX out of $CI_NODE_TOTAL 
    - echo platform: $PLATFORM  OS: $OS 

The above example will run a total of 9 jobs in parallel, each job will have different environment variable values as demonstrated in the script echo statements.  

You can also use the include attribute instead of script to trigger a downstream pipeline: 

deploy: 
  stage: deploy 
  parallel: 
    matrix: 
      - PROVIDER: aws 
        STACK: [monitoring, app1] 
      - PROVIDER: ovh 
        STACK: [monitoring, backup] 
      - PROVIDER: [gcp, vultr] 
        STACK: [data] 
  trigger: 
    include: path/to/child-pipeline.yml 

The above example will trigger the child-pipeline.yml 6 times for all combinations of values.  

See the trigger documentation for details on how to use downstream pipelines.  

GitLab’s solution is quite feature rich, the ability to use downstream pipelines allows for code reuse which can greatly help achieve build time performance boosts. While highly useful this still does not help to split up existing tasks to parallelize even further and achieve greater performance boosts, where again Incredibuild’s solution, which integrates well with any CI system, including GitLab, will help you split a single Make task to parallel execution. 

Comparison – Parallel CI

While all major CI/CD systems support some form of parallelization, we can see differences, with some systems providing distinguishing features and others providing only basic parallelization options. The following table summarizes the differences:   

Summary 

After reading this post you hopefully better understand how to use your CI system’s capabilities to parallelize. This allows you to scale beyond the limitations of a single machine and have the added value of the CI systems UI which shows the build jobs statuses and a simple declarative syntax for defining your pipeline.  

One important point to note is the inherent limitations of your build processes. At the end of the day, if your build process compiles a single binary in a single Makefile task, your CI/CD platform won’t help you to parallelize it. This is not something that is easily done by a CI system, which has no knowledge of the internal build execution. In order to optimize such use-cases, you’ll need something that tightly integrates with C++ build tools, such as Incredibuild’s solution.  

Another important point that was left out is artifact handling. In the examples, after you build the binary, you want it to be available for all the parallel instances which run tests on it. This is a subject for a future blog post, as each CI system has different features and usage for artifacts and there are also available options regardless of your CI system.