From zombie to cheetah: Undead Labs turns lumbering 1 hour lighting build times into 10 minute sprints
About Undead Labs
Undead Labs is an AAA game developer best known for its hit zombie survival simulation game, State of Decay, and now for the development of its much-anticipated sequel, State of Decay 2. The studio has a strong relationship with Microsoft and has signed a multi-title publishing agreement with the tech giant for the Xbox and Windows platforms.
Don’t let this studio’s name fool you. Much like the subjects of its hit games, Undead Labs is very much alive, kicking, and biting towards a glorious gaming future.
Working on a sequel to a successful first game is always stressful. Game developers want to outdo themselves so they can meet fan expectations and produce a title worthy of the IP. Making such a mind-blowing game takes talent, passion, and hard work, but it also requires something that can be a developer’s best friend or worst enemy: time.
Undead’s 55 developers, who are currently hard at work on State of Decay 2, needed time on their side. “Like most developers, we value iteration time highly,” says Ted Woolsey, General Manager at Undead Labs. Working on an AAA game also means a lot of time goes into getting the best graphics, and so Woolsey also mentions shader compilation and lighting data compute tasks as major time sinks.
Another challenge was found on the continuous integration optimization front, with the studio’s CI/build system cranking out more than a hundred daily builds of different sorts, including dozens of multi-platform (Xbox One and PC) code and content builds as well as a handful of scene lighting tasks.
“Incredibuild has provided a measurable quality of life improvement for the studio by speeding up our compute-heavy tasks and allowing everyone to spend more of their time actually making the game.”
General Manager, Undead Labs
How Incredibuild crunched it
Undead used Incredibuild to slash much more than zombies in half, by deploying Incredibuild agents on 3 dedicated build servers (12 core), and approximately 45 developer computers (4 core). The result: each and every developer on the team could scale his workstation up to the amounting 216 cores and harness that power for extra speed.
The studio’s developers utilized Incredibuild’s Dev Tools solution, which allowed them to accelerate a mix of custom build tools and save a significant amount of time on strategic time-wasting tasks, such as shader compilations and lighting middleware precompute tasks.
And so, Undead’s various build types saw a radical time improvement:
- For full code builds or cases where widely included header files are changed, build times went from ~45mins to ~7mins.
- For normal, incremental builds, Undead’s developers saw a 50% reduction on average. Since incremental builds are run many times a day by each developer, this was an utter game-changer.
- For shader compilation, build times went from a matter of minutes to a matter of seconds.
- The biggest consistent difference was found in Undead’s lighting middleware precompute tasks, which went from ~1 hr to ~10 mins for a given scene.
According to Woolsey, Incredibuild had a major impact on Undead’s CI optimization. “Yes,” he tells us, “Incredibuild decreases build times significantly, allowing for much faster iteration. Faster iteration allows us to distribute and test changes to the game faster, and in much more targeted increments. We believe that rapid iteration is one pillar of productive game development.” “In addition to code compilation, we’ve also found Incredibuild very useful for speeding up the process of compiling the many permutations of shaders we use as well as for pre-computing lighting data in our scenes.”
Now that Undead Labs’ developers got a taste of the change Incredibuild can generate on so many levels, they’re looking for even more tasks they can accelerate:
“We are sure as we get deeper into development that we will find additional uses for Incredibuild and we’re confident that it will pay for itself in a very short amount of time”.
The Bottom line
Lighting Pre-Compute Pass10 mins60 mins
Full C++ Source Code Build7 mins45 mins
Shader Compilation0.25 mins3 mins