Release managers today have their work cut out for them. They must adhere to increasingly high standards. The limits of speed are pushed beyond what was once thought to be reasonable and would have in the past been deemed impossible. At the same time, the bar has been raised when it comes to quality.
Now don’t get me wrong; I’m not one to argue against fast and frequent releases—the advantages are enormous. This has become the standard, and no organization can afford not to meet it. But that’s not to say it is easy to achieve. It is hard. Very hard.
That’s why releasing top-quality software fast enough requires release managers. They are the orchestrators of the entire software release management process and must understand it from top to bottom. Whether you consider the build process to be top or bottom, many of the biggest release management challenges start there.
Here, I’ll focus on three build-related challenges that require immediate attention. For each, I’ll describe the problem, explore the cause, and discuss how to resolve the issues to ensure the best possible outcome.
Challenge 1: Ensuring Build Quality
With the constant pressure for faster builds, how can release managers guarantee, maintain, and improve build quality? As we know, doing things faster does not translate to doing things better. On the contrary, increasing the speed of build cycles can backfire if quality is sacrificed.
One way to handle the trade-off between faster build cycles and quality is by reducing the human element. In recent years, advocates of CI/CD and DevOps have been promoting the idea that building an automated CI/CD pipeline will deliver better builds. Buzzwords such as “continuous code quality” are thrown out, promising the world.
While the continuous code quality approach might be too advanced for the current state of CI processes, when built correctly, a CI/CD pipeline can lead to significant improvements. By shortening the feedback loop and finding the problems faster, as well as minimizing the human factor (automated software quality gates are a big part of that), you can improve the quality of your source code.
As a release manager, examining the possibility of implementing a CI process is a good step towards quality. I know, no news here. But carefully implementing a CI process is not something that is widely spoken about, though it should be.
Automated processes can potentially harm build quality by delivering broken software at increasingly fast speeds. Sometimes deteriorating build quality is the result of resistance and hostility from team members that don’t understand the overall process. It’s common for build automation to generate broken builds, such as during the transition from a highly manual to a fully automated process. The problem is when nobody takes the time to fix them using reliable automated tests. As a result, bugs that would have been caught by experienced testers end up escaping into production. Another common hurdle is that proper build automation needs to be maintained by build engineers or a dedicated person for the job, who are often scarce.
The real challenge that release managers face is understanding that building an automated CI/CD pipeline is not an end in itself. Rather, achieving the best results requires a gradual and carefully planned transition process that takes into account the necessary balance between the needs of those individuals involved and the tools. Simply put, incorporating a CI process in your organization requires quality gates of its own, in addition to the code quality gates we all know and love.
The extent of automation is another delicate balance. In cases where a manual software release process is being implemented, release managers should examine this protocol closely and determine whether moving to an automated testing process would be possible and of benefit. Here as well, there’s a trade-off between automation, costs, and complexity, and each organization should determine the right combination.
There are other aspects of build quality that don’t necessarily concern the CI process, but as release managers, you can leverage the transition process to implement them. You know how it is; once you’ve garnered the support within your organization for such a big change, you can incorporate all kinds of best practices, such as enforcing code reviews (with in-line commenting) and performing post-release monitoring.
This might also be a good time for assigning responsibility for the build by recruiting a dedicated build engineer or promoting a suitable candidate to the role of build engineer and ensuring that they are compensated accordingly (now that the organization is committed to the CI process, securing funds might be easier).
It’s also not a bad idea to monitor quality metrics and indicators that point out bottlenecks within your processes. Not just the obvious metrics such as build failures and build time, but also the less common ones such as the most frequently executed and longest-running tasks.
Challenge 2: Incorporating Security into the Build Process
There’s a common perception that software security can slow down the build process. While there’s some truth to this, introducing security vulnerabilities into production software can create far more work and have lasting and unpredictable results.
A key aspect of ensuring build security is understanding that various parts of your build infrastructure could be insecure. First, your build server hardware and software might be compromised, especially if the servers are connected to the public internet to enable easy remote access. While there are legitimate reasons for wanting to access build systems remotely from outside your organization’s firewall, this can quite easily let in hackers.
Another issue is that development and build environments are often less secure than production environments. They may have very simple passwords—or none at all—and elevated privilege and permission levels that give users access to all systems and allow any type of executable to be run. Again, for a hacker, gaining access to your non-production environment could be just as valuable as gaining access to your production one. These problems can also serve as a backdoor into your source code and build systems.
In order to ensure build security, it is crucial to check that the development and test environments are as secure as the production environment. Developer privileges and permissions must also be strictly controlled. The good news is that issues related to infrastructure and build security are both easy to understand and to fix. The real challenge is educating your development and IT operations teams on secure coding practices, creating secure build and deployment systems, and general security awareness.
The bad news is that in practice, ensuring build security involves understanding the trade-off between convenience and efficiency. Though it is possible to improve various aspects of build security through automated build tools and processes—such as Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST)— they are not enough on their own. Like physical and general security, when it comes to build security the human element plays a key part and must be taken into account.
Challenge 3: Slow builds
With the growing reliance of businesses on software, time to market has become an increasingly high priority. Slow release cycles can certainly have a significant negative impact on the corporate bottom line. However, while slow builds are challenging in that obvious aspect of time to market, there are more “less obvious” hurdles slow builds present to release managers that affect their ability to do their job (or do it well).
You see, when builds are slow, they contain a lot of commits (because you can’t be building after every commit). So if the build breaks (a very reasonable scenario), the release manager (that’s you) has to figure out who broke that build. If you have a build with 20 commits, you’ll have wasted a lot of time going through 20 changes in search of the commit responsible. And it’s not just time wasted, it is also a lot of energy. Moreover, you’re left with no “clean” build to push to production. These challenges, as well as others, are discussed in more detail in this blog post.
So yes, in addition to ensuring that a build system produces high quality, secure software, release managers need to face slow builds.
The pressure to generate builds at increasing speeds may tempt release managers to skip various tests or other development tasks such as static code analysis. But this, of course, has a negative effect on quality, which leads them right back to our first challenge—quality. You see what I’m getting at. You just can’t win.
Some release managers might try to tinker with their build systems in order to fix specific problems. And as with the previous challenges, any simple solution will have unintended consequences. Why? It’s important to keep in mind that your build process or CI/CD pipeline is building solutions and not just source code.
Not only will most builds include contributions from multiple developers, but the code may also be written in multiple languages (e.g., JavaScript/Node, Swift, Kotlin, Python, Java, C#, or C++). Each language is used to write multiple types of applications, such as Client, Microservices, APIs, and Servers. Each individual language has its own environment as well as dependency and package management system (e.g., npm (Node.js), PiP (Python), Nuget (.Net), etc.). There is also a wide range of deployment options from cloud platforms, containers, and virtual machines. This results in a process that relies on fragile local toolchains with too many moving parts.
Sure, there are ways to restructure the code to support faster builds (such as utilizing PCH and forward declarations—for the complete guide click here). But this is quite complex and doesn’t always generate a desirable outcome.
Another way to go is to use distributed computing technologies, such as Incredibuild. Distributed computing allows build/release managers to optimize their build performance, reducing build time by up to 90%. These technologies do not require any source code change and no additional hardware.
By seamlessly distributing the tasks executed by the build across many remote cores in the user’s network or in the public cloud, build nodes or any workstation/virtual machine become a sort of “supercomputer,” with hundreds of cores and gigs of memory utilizing idle resources users already own in their local network/cloud. Incredibuild’s solution also offers an awesome build visualization tool that replaces your text output. Its intuitive UI allows you to track build progress; spot bottlenecks, dependencies, and errors; and replay your past builds. The subject of build analytics is an important one, and hopefully, I’ll have the opportunity to dedicate an entire blog post to it in the future.
Winning the Race
Release managers bear great responsibility, particularly when it comes to build management. Maintaining build quality, shortening build time, and ensuring its security affect the what (quality), when (speed), and the how (security) of the product. While these challenges can be tackled quite easily, this requires an in-depth understanding of the build process and knowing what’s out there to help.
As always, those who win the race aren’t the ones with the most resources, but those with the best information. Hopefully, this blog post has brought you closer to the finish line.
