Incredibuild Acquires Garden to Accelerate the Entire CI Pipeline. Together, we’ll speed up DevOps workflows, helping teams build, test, and deploy faster than ever. Read More

CMake Ninja Combo: The Gist

CMake_Ninja

Joseph Sibony

reading time: 

6 minutes

The word Ninja invokes the imagery of stealth and speed in my mind. When developing a “yet another build system” – as the creator of Ninja, Evan Martin, referred to Ninja in its manual – speed was of paramount importance. Naming the build system Ninja is apt as I will show you its capabilities in this blog post and highlight this USP. Let us begin.

What is Ninja and why another build system?

Let us begin with why Ninja was invented. During the porting of Chromium from Windows to other operating systems, build performance was becoming a bottleneck. You can read all about it here. Using Makefiles was found to be suboptimal and a new build system conceptually very similar to Make but with a focus on speed was conceived. It reduced the build start-up time of Chrome to under one second and was open-sourced soon after. It achieved this feat by:

  • Choosing speed over convenience as a design principle
  • Having a minimal set of policies or built-in rules (to be taken care of by ninja input files)
  • Builds are always being run in parallel, based by default on the number of CPUs your system has

Every build system eventually creates a graph of dependencies by parsing the build file. (In the next section, I have an image of this graph for Ninja itself). The build process traverses this graph to get the final output. In the case of Ninja, this is a two-phase traversal where:

  • In the first phase, the graph is walked up from the final output to the input files to see if there are any modifications and creates a plan for the build
  • In the second phase, the graph is walked down from the input files following the plan and executed in parallel

Further, the following low-level design decisions helped to make Ninja fast:

  • Re2c is used for parsing the build files which is highly efficient
  • Ninja canonicalizes the paths in the build file. Rather than treating the file paths as strings, it converts paths to a Node object, thereby eliminating costly string comparison for path equality checking. To compare if two paths are equal, Ninja just need to do pointer comparisons which are extremely fast
  • A build’s configuration (such as compilation flags used) is kept in a binary serialized format by Ninja. To determine if the output is out of date due to a change in the build configuration, a binary hash comparison is all that is required.

All these low-level optimizations make Ninja fast and efficient.

Getting Ninja

On Windows, building Ninja from source is easy. Here are the steps:

Install the latest version of Python and open a Visual Studio x64 native tools command prompt. Issue the above two commands, the first of which will download Ninja source from its GitHub repository and the second command will magically build Ninja.

Ninja is build using Ninja with a technique known as bootstrapping. The bootstrapping step first builds an executable called ninja.bootstrap.exe and a build.ninja file. This bootstrapping executable is further used to build ninja.exe. The build is quite fast as the dependencies for Ninja itself are quite small. I generated the following graph of dependencies of Ninja using commands:

  • ninja -t graph ninja.exe > graph_ninja.dot
  • dot -Tpng graph_ninja.dot > graph_ninja.png
ninja graph

Working with Ninja

To get a software build using Ninja, we need to create the input build file which by default is named build.ninja. Let us see a section of the build.ninja created for building Ninja as an example:

# This file is used to build ninja itself.# It is generated by configure.py.ninja_required_version = 1.3# The arguments passed to configure.py, for rerunning it.configure_args =root = .builddir = buildcxx = clar = linkcflags = /showIncludes /nologo /Zi /W4 /WX /wd4530 /wd4100 /wd4706 /wd4244 $    /wd4512 /wd4800 /wd4702 /wd4819 /wd4127 /wd4355 /wd4091 /GR- /wd4267 $    /DNOMINMAX /D_CRT_SECURE_NO_WARNINGS /D_HAS_EXCEPTIONS=0 $    /DNINJA_PYTHON="python.exe" /FS /Ox /DNDEBUG /GL -I.ldflags = /DEBUG /libpath:$builddir /LTCG /OPT:REF /OPT:ICFrule cxx  command = $cxx $cflags -c $in /Fo$out /Fd$builddir\$pdb  description = CXX $out  deps = msvcrule ar  command = lib /nologo /ltcg /out:$out $in  description = LIB $outrule link  command = $cxx $in $libs /nologo /link $ldflags /out:$out  description = LINK $out

Hand creating such a file is going to be a chore. Enter CMake, which has a backend generator specifically for Ninja!

Working with Ninja using CMake

Hand-creating input files for Ninja is difficult. A build generator system like CMake can be used to create the input files for Ninja. To showcase working with Ninja using CMake, let us build Ninja using CMake with Ninja as the backend.

Issue the following command from the directory where you have downloaded Ninja:

cmake -Bbuild-cmake -H. -GNinja

This should create a folder called build-cmake and you should find a file called build.ninja inside it. Here is the first section of build.ninja generated by CMake

# CMAKE generated file: DO NOT EDIT!# Generated by "Ninja" Generator, CMake Version 3.19# This file contains all the build statements describing the# compilation DAG.# =================================================# Write statements declared in CMakeLists.txt:## Which is the root file.# =================================================# =================================================# Project: ninja# Configurations: Debug# =================================================############################################## Minimal version of Ninja required by this fileninja_required_version = 1.5############################################## Set configuration variable for custom commands.CONFIGURATION = Debug# =================================================# Include auxiliary files.############################################## Include rules file.include CMakeFiles\rules.ninja

Make sure that ninja.exe created by the bootstrap version is in the path. Now issue the command:

  • cd build-cmake && cmake –build .

This should build Ninja using Ninja with a CMake Ninja backend. Ninja runs build commands in parallel by default, based on the number of available CPUs on the system. Let us modify the cmake build command to let Ninja display the build statistics.

  • cmake –build . — -d stats

Note: All parameters after — are passed to the underlying build system by CMake.

This prints us a nice build statistic:

Finished generating codemetric                  count   avg (us)        total (ms).ninja parse            2       1221.5          2.4canonicalize str        899     0.0             0.0canonicalize path       9473    0.0             0.4lookup node             1563    0.0             0.0.ninja_log load         1       51.0            0.1.ninja_deps load        1       32.0            0.0node stat               443     24.9            11.0StartEdge               72      62498.4         4499.9FinishCommand           69      2493.7          172.1CLParser::Parse         61      1157.5          70.6path->node hash load 0.72 (367 entries / 512 buckets)

I am reproducing an image from one of my previous blog posts that clearly shows how Ninja compares against Make on the compilation times of two open-source projects – bullet3 and LLVM.

Ninja - top open source build systems
Image source

Here is another study that showcases how Ninja is better than Make in terms of compilation times.

Conclusion

In one of my previous blog, I had described Ninja as a small open-source build system with a focus on speed. In this blog post, we saw how small the entire Ninja source is (from the dependency diagram we created using ninja -t command), and how easy is it to set up and work with it. We also saw how CMake works great with Ninja as CMake has inbuilt generator support for Ninja. It is for these reasons that Visual Studio 2019 has the default generator as Ninja for WSL2 builds. If you want a clean and no-fuss build system for your project do consider a CMake and Ninja combo.

c++ under the hood

Shorten your builds

Incredibuild empowers your teams to be productive and focus on innovating.