The future of software development is speculated by many.
There are countless articles out there on 2023 software development trends, most of them rehashing the same trends: AI, big data computation, low code development, IoT, CI/CD, cross-platform development, mixed reality (MR), and, of course – cloud computing.
I’m not going to discuss those… (except for cloud computing).
I want to discuss enterprise trends that relate to Agile.
Just to make sure we are aligned – Agile is not a trend.
Let me say that again:
Agile Is No Longer a Trend! It’s the Standard
Agile is the past, present and future of software development. In the 90s we had the software development crisis, where the time between the emergence of a need and its going into production was 3 years. The 2001 Snowbird meeting in Utah, that famous meeting of the agile community, kicked off many issues of fast delivery, though agile was yet mentioned.
Now that it’s almost 2021, agile is no longer just a trend, it is a fact of life. The future of software development is intertwined with agility. It was a slow process for enterprises, though. My frequent talks with enterprise dev leaders show that there isn’t a single enterprise that hasn’t already adopted or in the process of adopting agile practices.
The way it goes is that trends usually begin their journey with start-up companies, and once they have matured enough and proven their value, they are slowly adopted by enterprises. And while 95% of the start-ups today report (state of agile report) to using agile practices, adoption by enterprises runs at a slow but steady pace. It makes sense when you think about it, because well-established, big-software enterprises must go through a process that is ten times more complex than startups in order to achieve agility.
CI/CD is part of agile. When I’m consulting our top enterprise customers, there isn’t a single company in which continuous integration and continuous delivery is not the main part of their R&D strategy. Now I know that CI/CD is mentioned as a 2021 trend by a lot of the ‘trends’ articles out there, but I disagree. I think CI/CD has been around for a while now, and surely 2020 is all over it…
Sadly, more than half of the enterprises didn’t make it through the past two decades, and the main reason was a failure to take the digital transformation (DX) leap. And as much as adopting new technological practices is complex, the agile ship has sailed long ago.
Today, it’s a catch-up-or-die era. And obviously, everybody would rather catch up than die.
Out of the 12 principles of Agile software development, the first 3 principles are as follow:
- Gain customer satisfaction by early and continuous delivery – I think this is a real competitive advantage, meaning if you won’t practice early continuous delivery, your competitor will…
- Construct your dev cycles in a way that is flexible enough to support late development change in requirements.
- Deliver frequently means being competitive and improving continuously. Just think, a feature that is not in the release is a feature that was not developed.
The agile ex-trend reality leads me to these four trends I wish to discuss for the future of software development that enterprise organizations need to be aware of (2021, here we come!). These four 2021 software development trends, or what you may call predictions, are the main areas that require investment in order to adopt and maintain the main principles of agile practices.
They are both connected to and complement one another, resulting in fast-paced, automated, high-quality, and flexible software development.
Agile pre-requisites:
- CI/CD Automation
- Automated test coverage
- Fast dev cycles
- Frequent iterations
The ones to adopt these practices can usually experience positive results very quickly. Their time-to-market is much faster, which increases their competitiveness and customer satisfaction.
The ability to iterate frequently also translates to better quality products and more flexibility in accepting last-minute changes.
Advantages of agile for enterprises:
- Faster time to market
- Quicker update release (improved quality)
- More frequent iterations
- Better cross-company collaboration
We won’t cover ALL the trends related to enterprises’ agile adoption… But there are four important predictions and soon-to-be trends that I believe should be on the radar of enterprise dev organizations when planning for the future.
The trends I am about to cover can be divided into two megatrends which have already begun several years ago.
One megatrend is the shift-left trend.
Shift Left
The Shift-left approach is all about running your dev processes (software security for example) as early in the dev process as possible. It also involves not compromising on dev executions.
This could mean letting developers run tests before committing their code to the repository in order to get immediate feedback on their work correctness instead of failing a continuous integration build. Or running a code analysis test on every commit instead of only during weekends – so that developers won’t have to meet their code a week after they wrote it if it failed the weekend’s code analysis process.
Cloud Transformation
Another megatrend is cloud transformation – the process of moving your work to the cloud. While many organizations today speak ‘cloudish’ and 2010 is considered to be the Cloud Decade, and while predictions for 2020 were that 83% of enterprise workloads were expected to be in the cloud, and while COVID-19 essentially has a positive effect on cloud adoption, the cloud transformation is still not at its peak, but it’s definitely a trend.
Now, cloud transformation is a major trend for 2021 (Gartner forecasted that cloud revenue is expected to grow to $308.5 billion in the year 2021). It might have started to show several years ago (hence “the cloud decade”), but this year it’s about to finally peak. And cloud providers (AWS, Microsoft and Google) will peak with it (offering free budgeting to companies to test their cloud). Organizations (including governmental institutions) will not only speak cloudish, they will also act on it and understand what cloud transformation entails. The 10$B U.S. defense department deal with Microsoft Azure is a good example of this motion. Soon enough, when they call us to ask about Incredibuild Cloud, most of them Would have a clear cloud strategy and an understanding of their cloud transformation path, especially for their release automation and work-from-home challenges.
The following 3 problems are the main obstacles preventing enterprise organizations from being Agile. You can’t truly move the agile way without investing in all of them.
Challenges Enterprises are facing in adopting Agile practices:
- Code inflation
- Technical debt in testing
- Limited test-coverage automation
From nightly builds -> intra-day -> build ber commit
1. Problem: Test-Coverage Technical Debt
The code inflation in recent years was driven both by the high adoption of open source projects (yet another trend) as well as commercial libraries and custom code written en masse.
Code inflation leads to slower compilation time and to more code that needs to be covered by automated tests – whether functional, security, stress or otherwise. Having a massive amount of code added in a relatively short time usually means technical debt both in build times and test coverage.
The problem of technical debt around test coverage has many negative implications on the dev-release process.
1. Trend – AI-Based Test Generation Will Make Testing Faster and Better
To bridge this gap, companies started using all sorts of test automation solutions.
AI is another 2021 trend. But I don’t want to talk about plain AI, although it’s an interesting trend that involves a lot of money (see Gartner prediction regarding AI Investment at 2021). Instead, I want to discuss AI-based testing.
AI-based testing is an interesting solution for overcoming the test coverage technical debt, as it will allow covering this massive debt in a relatively short time and with reasonable investment. And here’s an interesting stat: at least a third of test professionals plan on using machine learning to make test automation smarter next year.
AI based-test generation can be anywhere from test suggestions to developers, faster test generation, code-less test generation and up to fully automatic test generation.
An interesting company to look into is Diffblue, which is a pioneer in providing an AI framework that autonomously writes test cases in masses. A few more companies and solutions are emerging (additional players are Testim.io, InteliTest, and PoniCode), and I believe we’ll see more coming up, as demand is very high.
Due to the nature of the C++ language, interpreting C++ code (as opposed to an intermediate language such as C#) is more complex and I believe that it will take more time until we see the dominant technologies mature enough to offer solutions for C++ code as well.
2. Problem: Full Test Execution Can Take Hours
Once we’ll meet our objective of having a high-quality test coverage automation, we’ll start experiencing another problem – the time it takes to run all these tests can be huge when testing an enterprise-scale software. As soon as testing automation is up and running, the testing hours an enterprise must commit to becoming endless (If we’re talking about inflation, that’s test inflation). Every update or change requires a test suite that can take a few hours.
On the one hand, we would like to test as frequently as possible, preferably per-commit, on the other hand, we can’t afford the time or resources involved in running a few hours of tests on each commit.
And even if we will run the full tests per commit, providing developer feedback only after a few hours of testing is counter to the Agile approach who is aiming for a response in a time frame of under 5-minutes…
A similar problem can be experienced with other time-consuming processes such as compilations, code analysis, and others.
The next two trends are targeting this exact problem and are aimed at allowing to shift-left these time-consuming processes with minimum compromises
2. Trend – Test Avoidance Tech Will Make a Dent in Unnecessary Testing
So you managed to solve the test coverage issue, but the number of tests is huge, especially in an enterprise-grade software that is investing in quality and automation. And it is not just the number of tests, it is also the various types of tests enterprises must commit to, they are endless. Here’s just a few of them:
- Unit tests
- API
- Integration tests
- Regression tests
- End-to-end tests
- Sanity
- Fuzzy testing
- DevSecOps
As an enterprise, you sometimes must minimize the number of tests without compromising quality. Every update or change requires a test suite that can take a few hours, so now what? Back to nightly builds?
Back to the slump?
It is a vicious cycle.
One solution is to avoid it.
Developers ask themselves all the time whether they must run a test on each change.
The answer is – they don’t, if they’ll know which tests to run…
Execution avoidance is all about avoiding running tasks that are not needed. Don’t run ALL the tests; only run the specific ones that you truly need. Although I am talking about test avoidance, compilation avoidance is also part of this trending technology. I realize that this approach takes getting used to. My instinct until recently was to test as much as possible, and Incredibuild helps a lot. By reducing tests and build times, you truly can test until you drop (and run builds frequently), in no time! But I must admit there are new approaches that are gaining traction, one of which is test avoidance.
Test avoidance is possible with companies like Sealights and Launchable that offer machine learning in test automation tooling to identify tests that matter. Understanding which tests to run to achieve the desired confidence level is exactly what companies such as Sealights and Launchable are here for.
The numbers are impressive. Here’s a chart of a success story from a leading car manufacturer, showing that running the right 20% of tests with Launchable leads to the same confidence level as running a randomly selected 75% of the tests. Amazing, isn’t it?
This has led to undoubtable ROI and faster time to market:
In the meantime, until Kohsuke’s solution will be available for all, and in the cases where running 25% of the test cases can still accumulate to a few hours of executions, enterprises need the speed to allow running all these processes without compromising on speed, quality, and execution frequency – this is where Incredibuild can really help, reducing tests time significantly.
3. Problem #3 – Slow Build Times and Infrequent Executions
As I previously discussed, long build processes are one of the main obstacles for Agility. Whether these are long C++ compilations executed by the developer, or build servers, or other time-consuming development tasks.
Compromising on build frequency and not running a build per commit or compromising on what to run per commit leads to finding errors late in the process which causes a long and tiresome investigation of which commit broke the build, and the inability to predict or have a releasable version on a daily basis.
We must make sure our tests, compilations, and other compute-intensive processes do not drag our enterprise back to waterfall.
In today’s world of High-Performance Computers, clusters, and distributed computing, having time-consuming processes due to limited compute power is a problem that belongs in the past.
Trend #3 – High-Performance Computing
The new AMD threarippers have disrupted the market with affordable many-cores computers, (just think of a 64-Core desktop Processor that delivers incredible performance for many demanding workloads), and distributed computing allows companies to easily use hundreds and thousands of cores to run their compilations, tests and other compute intensive processes.
I believe in big infrastructure solutions, and more hardware just doesn’t cut it. With distributed computing, there’s no limit to the infrastructure you can utilize.
Take Minitab for example: they have a great test coverage, fitting the quality-grade software they produce. But this quality comes at a price – an 11 hours-long test cycle. Upgrading their build servers from 32-cores to 64-cores will not do much to reduce these test times to what they need to be Agile and competitive (apparently, 11 minutes is the magic number). With distributed computing, Minitab can seamlessly scale their build server to harness hundreds and thousands of cores to distribute their tests to – just when they need them. These can be either idle cores of hosts in their local network, or on-demand cloud resources, even cheap Spot instances that automatically scale up and down according to your dynamic needs.
As Incredibuild’s CTO for the past 10 years, I see enterprises in a constant chase for compute power when they already own thousands of idle cores across their network with which they can scale each of their host to 1,000 cores super-computer – if only they could tap these resources. With distributed computing, they can!
While testing is at high focus, it’s not the only type of process in an enterprise CI/CD that causes builds to be very long. There are other processes that are bound by compute power, such as compilations, code analysis, asset creation, code signing, packaging and more.
Another last trend I believe has matured enough and I thought worth mentioning in this post is the cloud-managed CI/CD solutions.
4. Trend – Managed CI/CD in the Cloud Will Automate Release Pipelines
A last trend I believe has matured enough to be considered a trend fitting enterprises and I thinks is worth mentioning, is the cloud-managed CI/CD solutions.
OK, so cloud transformation is a trend and CI/CD is a disputed trend (a “has been” trend, in my opinion), so why not join forces and run CI/CD in the cloud? After all, it has many perks.
The benefits of CI/CD in the cloud for enterprises are around agility, scalability, self-service, maintenance, security, as well as business reasons such as companies moving their budgets to operational costs rather than capital costs – meaning, for example, purchasing services on a predictable yearly basis as opposed to purchasing hardware that should be used for a few years.
Now, let’s stop and discuss the tools needed to reach full CI/CD.
Major cloud vendors such as Amazon AWS, Microsoft Azure and Google Cloud Platform, along with other vendors such as Jfrog, CircleCI and more, are fighting around the dominance of offering an end-to-end managed cloud solution for CI/CD and build pipelines.
These tools were honed and matured for the startup ecosystem. Managed CI/CD was first adopted by startups, mainly in areas where frequent releases are a must-have, such as web applications, cloud services and mobile apps.
Nowadays, after these services have matured enough, we see these practices applied to the big-enterprise world, even if in a slow pace. Some of our largest enterprise customers are already transitioning their on-premises CI/CD to a fully managed cloud solution. Although there are still some concerns around placing the organization’s source code on the public cloud, outside the organization network, looking back to just a few years ago, companies were reluctant to move their business data and customer details to the cloud and share their source code with third-party vendors, but today, most enterprise companies are using services such as Salesforce and SAP that hold their entire business data in the cloud.
Additionally, Cloud-related technologies are trending not only around managed CI/CD services, but in the entire dev eco-system; Repositories in the cloud, remote builds for developers, virtual desktops and SaaS solutions for testing are just the tip of the iceberg and are marking the path the software industry is taking.
Additional cloud-transformtion solutions:
- Cloud repositories (GitHub, GitLab, Perforce cloud)
- Cloud builds for developers (GitHub codespaces)
- Virtual Cloud desktops
- SaaS testing – Applitols
- Work from home bandwidth optimization – Teradici
- And more…
Before You Push Your Enterprise Over the Speed Limit…
Assess Your Resources and Priorities
Every digital/innovation/technology decision-maker in every enterprise must understand the maturity and stage of the company he’s serving.
After the COVID experience – every organization must strategize their move to the cloud, continuous automation, collaboration, and work-from-home enablement.
The best example is game studios: having already invested heavily in automation, the world’s leading game studios managed to not only release new games, updates, and fixes during the pandemic’s critical stages, but they were even able to grow their business during these times.
Every organization must understand its limitations, its strategy, and how it can push forward towards automation and ever-improving agility.
Otherwise, they risk joining the leading enterprises from the year 2000 who could not meet the competition of the fast-pacing startups that replaced them.
Being an enterprise doesn’t mean being slow, it means we have more resources and bandwidth to facilitate innovation faster.
Understand your gaps and fill them fast.
