Frag Lab

About Frag Lab

Frag Lab is a Ukraine based indie video game developer, comprised of a team of developers who are all passionate about developing the best AAA online F2P FPS games. As fans of the genre, and with experience working on noteworthy FPS titles in the past, Frag Lab is shaping up to be a name shooter game lovers will hear about a lot in the near future.

The team develops the C++ game client on the Windows platform with the Visual Studio.

The game server, on the other hand, is developed for both Windows and Linux OS. The Windows version of the server is used because of the convenient Visual Studio IDE while the Linux version is built with Clang and aimed for production deployment.

The Challenge

“The top challenge for us is compilation times of local changes for developers,” says Frag Lab’s Sergey Rustamov. “Due to the way the source code is organized and C++ language itself, many changes especially in common C++ header files trigger a massive rebuild in different places. Without a distributed build system, this operation is extremely time-consuming. As a side effect the overall CPU usage is so high during the compilation process it makes it almost impossible to switch to another task, so the developer’s time is just wasted.”

Slow compilations are not Frag Lab’s sole performance challenge. The studio’s continuous integration system, powered by Jenkins, needs to act as fast as possible.

“Our development philosophy is built around the idea to fail as early as possible,” shares Sergey. “We prefer to make mistakes in the development stream and prevent it from the mainline one. The speed of our continuous integration system is a critical component of this idea that enables us to fail early on submit to perforce and allow the QA team to get autobuild and fail again almost immediately after a developer made a change in the source code.”

How Incredibuild crunched it

With Incredibuild, each of Frag Lab’s developers can distribute the various tasks related to the game’s development to other machines in the network, and enjoy the speed that comes with the computational power of the sum of all the machines’ cores.

Incredibuild is used for the following:

• Helping developers to compile local changes quickly
• Speed up our continuous integration system to reduce waiting time for build and autobuild artifacts

All of Incredibuild’s build steps are managed by a build system called WAF. It integrates with Incredibuild directly using the Make and Build solution, which allows developers to accelerate tools like WAF, Ninja, MS Build and many more out of the box.

The Linux version of the game server isn’t distributed but scaled vertically on all available CPU cores to speed up the compilation process.

When it comes to speeding up the CI cycle, Incredibuild’s distributed processing technology distributed and thus accelerated the following:

• Autobuild from development streams triggered by submitting actions
• Canary autobuild for mainline to detect broken main build early
• Build from mainline and release branches

The Bottom line

When asked to sum up his experience with Incredibuild, Sergey tells us that “from a developer’s perspective the difference between build time with and without Incredibuild is huge. We collect build metrics from the development team to see the average time for most the typical build actions and it takes in average 200 seconds to build a local change, while without Incredibuild it varies from 15 minutes up to several hours depending on how massive change is. Incredibuild is a key component that provides us with fast CI pipeline, especially for autobuild flow.”