Survey Report - 2022
2022Big DevBuild TimesRead the overview
Introduction & Methodology
We launched this survey with a single question in mind.
What is the true state of the software build process today, and what impact does it have on the broader business?
To uncover the answer, we spoke to respondents across a wide array of industries, roles, and company sizes, and with varied amounts of hands-on vs managerial experience. We asked them whether they suffer from long
build times, and whether this problem is getting worse year over year. We delved into the frustrations they face, the technologies they use, and how much time and
productivity they could regain if they had the right processes and tools in place.
The results reveal the nature of the build experience for today’s enterprises and paint a clear picture of how organizations can translate faster build times into true ROI.
To collect these insights we commissioned a survey of 300 senior managers or above from North America, APAC, and the EU in companies with 1,000 employees or more that implement C++ software projects. The survey was completed by Global Surveyz, an independent survey company, and took place during Q4 2021. The titles of our respondents ranged from CTOs to software engineers and architects, as well as firmware engineers, embedded systems, build/release developers or those in configuration management.
The respondents were recruited through a global B2B research panel and invited via email to complete the survey. The average amount of time spent on the survey was 5 minutes.
The answers to the majority of the non-numerical questions were randomized to prevent order bias in the answers.
Today’s builds take a long time,
and it’s getting longer
Running a build takes an average of 20 minutes, with almost 25% of respondents spending more than half an hour on each build. For CI/Release builds, this percentage jumps to 32%. Respondents spent an average of 57 minutes waiting for builds to finish. This number is rising, with 96% of respondents saying build time increased an average of 15.9% in 2021.
Long builds have a direct impact on productivity,
frustration, and costs
Time spent on a build is time not spent on testing, iterating or being productive. 98% of respondents say they are wasting time waiting for builds to finish. 91% consider this a problem, and slow build times come in third in the top frustrations experienced in day-to-day development. Almost half of respondents are using the cloud for development and deployment, where slow build times equate directly to more compute utilization and higher costs.
The most commonly used tools are making
build times longer
76% of respondents use static code analysis, with an additional 11% planning to start in 2022. 88% of respondents use dynamic code analysis. These tools are overwhelmingly in use, and are required to increase code quality and shorten development cycles. However, they do not come without a cost to build time. This may cause frustration over long build times to become even more prominent.
Current solutions to the growing
problem aren’t sustainable
Organizations which successfully decreased their build times focused on better hardware (44%) and decreasing codebase size (33%). However, adding hardware is not a sustainable fix to growing build times, and is an extremely costly and limited solution. Decreasing codebase size might help at first, but once code is in good shape this also has limited impact. Organizations need to prioritize managing their hardware better over band-aid solutions.
The impact of the right tools and processes
could be game-changing
Putting the right tools and processes in place would add 65 minutes of productivity on average every day, with this number increasing to 85 minutes per day for R&D, and 80 minutes per day for DevOps. 65 minutes per day is 36 days a year, which on a $60/hour developer salary is a $20,000 saving overall. Productivity savings could be used to hire more staff, implement best practices, debug faster, or boost customer experience.
State of Development
Build Times and Challenges
Time Spent on Builds and CI/Release Builds
Running a build or a CI/Release build takes its toll. Companies are spending 20 minutes on average to run each build (figure 1) and 23 minutes to run a CI/Release build (figure 2).
69% of companies are spending 10 or more minutes to run a build, and 24% of companies are spending more than 30 minutes (figure 1).
82% of companies are spending 10 or more minutes to run a CI/Release build, and 32% of companies are spending more than 30 minutes (figure 2).
Instead of being able to build quickly, test and get results, builds are taking a long time, slowing down the whole development cycle and having a direct impact on productivity.
Weighted average: 20.3 (Min)
Weighted average: 23.5 (Min)
Time Spent Per Day on
Waiting for Builds to Finish
98% of respondents admitted they waste time waiting for builds to finish (figure 3). The average time spent waiting for builds to finish is just under an hour at 57 minutes (figure 4). It varies by industry, with defense/military holding the slowest build times (72 minutes) and banking the fastest build times (38 minutes).
However, even 38 minutes is a lot of wasted time in terms of productivity. In many cases developers need to wait for builds to finish before they can move onto the next task, so this wait is literally time down the drain.
Increase in Build Times 2020-2021
(and What Decreases Build Time)
96% of survey respondents said build time increased by an average of 15.9% in 2021 compared to 2020 (figure 5).
This can probably be attributed to increase in codebase size.
Only 4% reported that build time decreased. The top reasons they give for the decrease are better hardware (44%) and a decrease in codebase size (33%). Better hardware might seem like a quick fix, but it is a very costly solution which organizations will pay for whether it’s utilized or not. It’s also not sustainable, as a team can’t keep adding more hardware within a single machine to meet growing needs.
Decreasing the codebase size might be helpful at first, but will provide diminishing returns over time, and have marginal impact once an organization has made initial fixes.
Weighted average: 15.9%
Development Technologies in Use
The most used technologies and methodologies for accelerating or improving the development cycle are dev in the cloud (49%), static code analysis (46%) and dynamic code analysis (28%). Just 8% of survey respondents have no plans to use dev in the cloud, likely because of security/compliance issues.
About one quarter (23%) of respondents plan to start using build acceleration in 2022.
Are Slow Build Times A Problem?
Analysis by Industry
Slow build times are a problem for 91% of companies. Only 33% said this isn’t a big problem and 58% said it’s a significant problem or a problem to some extent.
When comparing by industry (figure 9), defense/military suffers the most from slow build times (73%), followed by IT services (67%) and automotive (63%).
Top Developer Day-to-Day Frustrations
97% of survey respondents report that they experience frustrations in their day-to-day work.
The most frustrating issues for respondents are analyzing crashes from production (35%), learning new technologies (35%), and long build times (31%). While crashes and new technologies are somewhat necessary evils, long build times can be solved with the right technology.
Static code analysis is one method of reducing crashes, but this adds time to builds, which is likely to increase frustration overall.
This requires a solution since adoption of both static and dynamic code analysis are growing in the market (figure 7) as methods to improve testing and quality.
Top Time-Wasters According to Dev Teams
The top time-wasters are creating internal documentation that no one will read (82%), unnecessary changes in requirements (78%), and slow builds (69%). Also on the list are slow tests, highlighted by 67% of respondents. Slow builds and slow testing feed into one another. If a company can speed up builds, they can speed up tests.
Top Requested Areas to Reduce
97% of respondents admit they have areas they wish they could improve in their day-to-day work which would save on development time.
The top requested areas are moving to new build systems (35%), optimizing build scripts (31%), and being able to accelerate their build times (28%). All of these top requests are aligned with improving build times, showing the significance of this need.
C++ Codebase Size and 2022
91% of survey respondents have hundreds of thousands of lines of code or more. 76% have millions of lines of code or more (figure 13).
Codebases are expected to grow by 18.6% in 2022. This will grow most at large companies with 5K+ employees (21.6%), while in companies of up to 5K employees, it is expected to grow 16% (figure 14). Bigger companies face bigger problems.
As codebases grow, build times will get even longer.
Impact on Productivity of Having
the Right Tools and Processes
95% of survey respondents indicated that with the right tools and processes, they can save (average weight) 65 minutes per day. Based on 261 working days per year (in 2021) and 8 working hours per day, that translates into approximately 36 days per year. Translating this into cost savings, 36 days is almost $20,000 per year if we take $60/hour as a typical dev cost. Productivity in this area could realize serious ROI even for those who say it will only save 30 minutes or an hour per day. Saving two hours per day would mean a company could add another full-time developer to its team.
Faster builds also allow the adoption of best practices such as feature flagging, working in parallel on a few code branches and more, and allow for faster bug resolution, faster time to market and better customer success opportunities – all of which impact an organization’s bottom line.
Weighted average: 65.6 Minutes/day (36 days/year)
Impact of Having the Right Tools
and Processes on Productivity by Role
When looking at the impact on productivity of having the right tools and processes by role, we see that while the average time saved for all respondents is 65 minutes per day. The number grows by 30% when it comes to R&D, and the right tools and processes will save them 85 minutes per day.
In DevOps and R&D there are much higher issues with recruitment, so when translating the time savings to cost savings (figure 16), it’s clear to see how impactful the right tools and processes could be on these teams.
Cloud Usage throughout Development Lifecycle
The cloud is already used throughout the entire development lifecycle. The top stages for using the cloud overall are in development (43%) and deployment (42%). When looking at the industry breakdown, banking has the highest rate of cloud usage. When we connect cloud usage to slow build times, we can see a direct correlation to costs. Time-consuming builds cost much more on the cloud in terms of compute utilization.
In some industries, it’s impossible to use money-saving options such as spot instances, and therefore costs can spiral quickly. Organizations need to look for ways to keep cloud costs low while optimizing for faster builds.