THE COALITION

This case study has been co-written by Microsoft and Incredibuild. For the version published on the Microsoft site, click here.

About The Coalition

Microsoft’s The Coalition Studio has chain-sawed its way into the forefront of AAA gaming. As its name implies, The Coalition is dedicated solely to developing, supporting and innovating one of the top FPS brands, Gears of War, after Microsoft bought the franchise from another Incredibuild customer, Epic Games.

Located in scenic Vancouver BC the studio’s 200 employees have worked hard to immerse millions of gamers into the post-apocalyptic planet Sera with the record-breaking Gears series on the Xbox One and Windows 10 platforms As long-time Incredibuild users, the Coalition’s developers have integrated it with Microsoft Azure to seamlessly scale to hundreds of additional cores through a hybrid cloud environment.

The challenge

2016 proved to be an amazing year for Gears fans everywhere. With the release of both Gears of War: Ultimate Edition, a high-definition remake of the first game in the iconic series, and later Gears of War 4, the next chapter in the FPS saga, the hype was real. The Coalition’s developers needed to deliver on that hype and overcome external and internal production challenges. Internally, frequent and massive use of Unreal Engine 4 (UE4) presented significant challenges, as told by The Coalition’s IT Manager, Joe Vogt. “We use Epic’s Unreal Editor 4 as a primary tool for content creation such as levels, characters and vehicles.

The editor needs to compile all shaders beforehand, opening a level can take up to 30 minutes in some situations.” With more than 100 team members using UE4 on a daily basis, a solution was needed to accelerate the shader compilation times. “You start doing the math and find out that with 30 minutes just to open a level, 4-5 times a day, you can spend a third of your day doing nothing,” says Vogt. Externally, several 3rd party outsourcing partners worked alongside The Coalition to provide the studio with augmented production capacity.. Some of these partners were smaller organizations that did not have the benefit of dedicated Incredibuild machines needed to compile shaders and effectively utilize UE4. This situation left these partners in need of a solution.

How Incredibuild crunched it

As a 1st party studio inside of Microsoft, The Coalition Studio is able to use and benefit from the Azure cloud computing platform. Microsoft Azure provided The Coalition Studio an easy and effective way to deploy hundreds of dedicated CPU cores to be used for accelerating local shader compilations. The Coalition Studio harnessed Incredibuild’s unique process level virtualization technology to scale Azure performance beyond the capabilities of a single Azure Virtual Machine.

The Coalition Studio deployed dozens of 64-core Azure Virtual Machines (VMs) running Incredibuild which were made available as a “virtual supercomputer” to local artists and animators who needed to use these resources for shader compilation. By harnessing the power of Azure VMs, Incredibuild creates an “on the fly” virtual environment where all those cores are working together for a single process execution. Thus, scaling the platform to meet very large process execution requirements and seamlessly accelerate UE4 shader compilation and level loading times for developers, programmers and designers.

Internal Acceleration: Scaling Azure Within the Company

The Coalition Studio spent almost three years creating Gears Of War 4. During this time, local, dedicated incredibuild resources were used on-premises to accelerate UE4 performance as well as code compilation. As is common in large scale game development, the final six months of production becomes “crunch time” as the studio pushes hard to ship the game on time. During this critical phase, The Coalition Studio needed a way to augment Incredibuild resources without adding more on-premises equipment.

The Coalition Studio was able to deploy a large Azure-based Incredibuild farm with connectivity provided by Microsoft Azure ExpressRoute circuits.

This allowed the Azure based agents to appear “local” to TC’s existing production coordinator. 700 cores were deployed in this fashion and used throughout the final production push for Gears Of War 4, resulting in a significant reduction of both shader and code compilation times.

According to Vogt, “Incredibuild absolutely impacts our ability to iterate and move forward. Our iterative cycle includes creating a build of the game, playing it, finding bugs, reporting the bugs, killing those bugs and the producing yet another build, sometimes two times a day. Rinse and repeat. It’s all about how fast can a user get the latest version of all the data and get back to work. Without Incredibuild, our development would grind to a halt.”

External Acceleration: Granting Small Partners Much-Needed Juice

When it came to external partner productivity, Azure and Incredibuild resources were outsourced for a short but crucial boost to empower compute-challenged partners.  “In one instance,” recalls Vogt, “the partner was unable to provide any dedicated IB farm capacity onsite which caused huge waits for users when opening massive Gears of War 4levels in UE4.”

Again, The Coalition utilized the flexible nature of Azure consumed services and Incredibuild’s parallel computing tech in one package. The only difference is this time, it granted outside partners the power to scale their performance as well as enjoy unparalleled acceleration, no matter how limited their initial resources are.

“We deployed around 160 cores to a public-facing Azure datacenter and configured a site-to-site VPN with the partner location. This allowed the small number of local Unreal Engine users to benefit from the Incredibuild farm capacity in Azure to accelerate their work.”

Since this was a short-term engagement, like many other micro- tasks that are part of developing a game, The Collation enjoyed Azure consumed services through hybrid cloud bursts for the limited time they were needed to boost Incredibuild, instead of buying expensive hardware which would overstay its usefulness.

The Bottom line

The Coalition seemed to do the impossible by releasing 2 AAA games in one year. They were able to pull it off in part by:

• Scaling Azure to its max potential with Incredibuild: With Incredibuild’s power creating a 700 core Azure virtual super-computer, well beyond the limitation of Azure’s most powerful single virtual machine, The Coalition scaled to the sky for maximum acceleration.
• Eliminating unnecessary time-sinks: Backed by Azure compute power, Incredibuild dramatically cut shader and code build times, transforming UE4 level load times and  Allowing production staff to keep working instead of waiting.
• Accelerating outsource partners:  Limited partner resources didn’t pose an obstacle since The Coalition allocated Incredibuild operated Azure cores to help under-powered external companies.
• Saving resources with flexible deploying: Real-time scaling up and down for both internal and external needs allowed The Coalition to allocate solutions for any period, and just that.