Nobody wants to be caught off guard by an exposed security vulnerability. According to the 2020 Data Breach Investigation Report from Verizon, data breaches due to poor security practices affected approximately 41% of the almost 58% of companies who fell victim to cyber-attacks. Knowing this, the practice of DevOps Security is crucial in preventing loss of revenue or even a portion of a firm’s market share.
Furthermore, the hyped focus on DevSecOps shows that the market will grow from $1.47 billion as it was in 2018 to a whopping $13.63 billion by 2026 according to research completed by Data Bridge Market Research. This shows that DevOps Security is quickly becoming a top priority, and it is happening in addition to the paradoxical desire to have software delivered more quickly while also maintaining or improving quality.
A poll of developers shows that 72% feel their organization’s security is either “good” or “strong.” This is up from 59% the prior year. Additionally, the constant investment in DevOps Security and interest in acquiring companies with a DevSecOps focus (see Jfrog’s acquisition of Vdoo) is clearly a sign of more focus on the technology in the future.
Now we are seeing major providers like GitHub doing security testing as part of their cloud-based services. They employ “Semmle”, a semantic code analysis engine that allows for querying for specific patterns. And, those features not provided natively are usually handled by third-party vendors via each service’s individual marketplace. On the other hand, tooling aside, data from an e-book available from Vectra AI shows a lack of formal sign-off on new production-ready software versions is one of the most problematic issues for a holistic security approach.
In this article, we’d like to focus a bit on the discipline of DevOps Security and some best practices used to help with DevSecOps to overcome these challenges. We will also address some of the myths around the subject. Finally, we will talk about what the future may hold for the practice of DevOps Security.
How did we get to DevSecOps?
The simple answer is “out of necessity.” The speed at which software and application development moves has increased over the past years. Much of this is thanks to the automation put in place by DevOps engineers that handles many aspects once completed by multiple subject matter experts. But, the increase in velocity has a flip side.
If there are no DevOps Security practices in place, or the practices are lacking or inappropriate, the increase in velocity may provide an opportunity for defects to pass through. Defects are a normal part of software development. Those centered around security are among the most critical to recognize early in the development process. This is where the practice of DevSecOps shows its usefulness.
A review completed in 2020 on “The State of Modern Applications in the Enterprise”, reported that 78% of respondents listed security as being a key priority. This focus on aspects of security through the entire process assists the developers, stakeholders, and DevOps engineers to work towards forward-thinking in terms of DevOps Security. Which is related to constant funding and significant new investments in DevOps Security.
How does DevSecOps compare to traditional DevOps?
For one, DevSecOps can be thought of as practicing DevOps with an increased focus on security through the entire development process. DevOps with a security add-on, if you will. But further than that, everybody is involved in helping maintain the security of an application or service. Developers have ways to help prevent code from ever reaching the codebase and therefore, production.
To be successful in DevSecOps you need to start pushing the idea of “Shifting Left” with security. Look at the technology for git pre-commit hooks. It is one way that developers can check their local code against a set of policies you create on your repository. If it isn’t a “clean commit,” it won’t be accepted. To “Shift Right” is just as important. Meaning you have a view into the internals of a production system, with appropriate monitoring and alerts that show on data points and at thresholds you configure, including security-related events and information.
This newfound mindset in the DevOps world is creating more and more opportunities for teams to work closely together. Keeping the product free of as many security incidents as possible and maintaining data integrity. Working with automated QA in a CI/CD pipeline with a focus on security aspects is one way QA adds to the equation. Detecting a vulnerability early and with automation is just another way DevOps Security continuous software delivery in a CI/CD workflow.
What DevOps Security is not.
It is not a replacement for in-house subject matter experts on network and data security. It is meant to complement their work.
You cannot get to a place of appropriate DevSecOps just by tooling alone. While there are many excellent tools that are used for configuration management and coordination of safe software delivery, they should not be considered a “silver bullet.” True, there are many tools that are geared towards automating many aspects of DevOps Security. These tools are only part of a bigger picture.
DevSecOps challenges in the real world.
There’s always the aspect of how change affects the workflow of a team. Adding the DevOps Security aspect to an already established team can pose a few challenges. Some of which have to do with the culture and organizational aspect. Culturally, engineers don’t see additional steps as a positive thing. However, when implemented in a way that shows the value we can see a path for it to make its way into practice.
As an organization, predictions from analyst firm Gartner says that 75% of DevOps initiatives would fail to meet their goals. Much of this has to do with ongoing issues around learning new technology along with the normal toil of change. It is important to note how the recruiting aspect is also affected. With DevOps Engineers already in high demand, some engineers are put into the role without substantial base knowledge. This creates a situation that may be less costly but can cause a negative impact on the software release schedule.
Compromise between speed and security is the ultimate goal. We don’t want to slow developers down, but we do want them to be mindful of ways to prevent security vulnerabilities much earlier in the process (shift left, people). Not just developers, but everybody on the team. There are those that have knowledge of internal systems that may need to share that tribal knowledge.
While most DevOps engineers are quite familiar with any “legacy” services in their environments, they may not have that tribal knowledge to understand security aspects around its usage. This is an opportunity where collaboration is key to understanding the flow and any potential scenarios. Change can be fast. Especially with the ability to use automation to create cloud infrastructure to deploy simple containers.
A challenge in themselves, containerization and cloud infrastructure are part of the new landscape that is also under our charge. Containers may require additional scanning to ensure they are in parity with policy. Cloud infrastructure is often more exposed than on-premise installations. The security of these resources is just as important as the data center infrastructure that was and still is commonly seen for those offering their services online. Ways to ensure these follow DevOps Security guidelines is critical to moving forward with today’s tech.
DevSecOps rightfully taps the brakes to allow for a full view of the security aspects of a deployment or infrastructure change. Even the most trained developers aren’t security experts, and can unknowingly create additional security debt for items in code. Finding developers that already have a mindset on security vulnerabilities can be tough. There is not a lot that has the new security skills to hit the ground running. Still, many are understanding the need and are promoting the use of tools and services that help with securing aspects they are in charge of.
Let’s not forget how the new distributed workforce has created its own set of challenges to security. Large teams sharing information over multiple avenues can create new areas of concern. More can be learned about this through up-and-coming data we are seeing on the subject. Read more from Syntax’s Mike Rulf, who recognizes why security has become more important than ever.
This is true for infrastructure just as much as it is for the software itself. During the architecture of the application or service, security policy must be a priority over just “making it work.” While it may have been acceptable to add technical debt in the past, doing so for security items is not a tenet of good DevOps Security. The items that make up an environment must be tested just as much as the services running in it. This means taking into account:
- Using the least amount of privileges for user or service accounts.
- Code reviews that include security aspects.
- Keeping apprised of the latest coding standards for security.
- Network segmentation and edge-device vulnerability testing.
- Load testing to understand how the underlying resources hold up.
The tools of the DevSecOps trade.
We touched on some best practices for DevOps Security, now we will talk about some of the tools used in today’s infrastructure and application deployment pipelines. Consider the tools used in today’s Kubernetes clusters that help maintain specific boundaries for the containers being created within.
Open policy agent is used as an admission controller whose job is to enforce specific policies. There are many advanced features that are not readily available unless some sort of controller is in place. Most any aspect of image rule enforcement at the start as well as manage any incoming objects.
Tools like OPA used to enforce policy along the way to production is one part of practicing Policy as Code. Some of the biggest names in multiple industries rely on this type of technology to help secure their systems. Here are a few notable ones that have shown themselves as adopting PaC:
- Atlassian
- Capital One
- Chef
- Goldman Sachs
- Intuit
- Netflix
- SAP
- Tripadvisor
- Yelp
Integrating automation like SonarQube for CI/CD and git pre-commit hooks can take the collaboration between Development and Security Engineers to a measurable and actionable level. Meaning policies and scenarios specific to your industry can be defined on a case-by-case basis to cover various threat models. From the collaboration completed around possible threats, thresholds and actions can be set for alerts or action items for automated recovery.
Beyond just DevSecOp tooling.
A more complicated and in-depth workflow can be modeled to include some of the critical aspects of DevOps Security:
- Integrated tooling – Tools that help identify policy and security items in code, during deployment, and post-delivery.
- Continuous Feedback Loops – Used to automate feedback to developers and experts to understand what happens when code runs correctly as well as when things aren’t looking right.
- Secret Management – There are many products meant to secure shared passwords and other information. Secret management for decentralized work is critical for smooth operation.
- Threat Analysis – Determine what types of data are available that need to be secured along with edge servers or other endpoints that provide a means to cause a disruption.
- Limit Admin Rights – Administrator rights are some of the most widely misused permissions available to us. Using the least amount of permissions required is best to ensure the stability of the environment.
- High Prioritization – Items regarding security must be put at the top of the list in order to truly support a DevSecOps model.
- Change Management – Using a means to control change often requires a change management process. This could be a highly developed workflow, or something as simple as a release checklist.
- Key Performance Indicators (KPIs) – In order to know if your efforts are paying off, it is important to define KPIs that can be tied to the business objectives. Defining metrics to show a baseline and decrease of security-related incidents is an easy way to demonstrate the need for DevOps Security.
Keeping a constant eye on the latest news in security is also critical for success. There are many resources that are designed to keep architects, developers, and IT specialists informed of the latest threats. Finding and sharing this type of information with the team will help prevent incidents before they even start. For example, Qualys is taking Infrastructure as Code (IaC) to new levels by integrating the shift-left approach to infrastructure. Their new offering, CloudView, “Adds Security for Infrastructure as Code Enabling DevSecOps Teams to Start Secure and Stay Secure.”
- Codacy
- SonarQube
- Acunetix
- Logz.io
- GitLab DevSecOps
- Contrast Security
- Aqua Security
- Digital.ai
- WhiteSource
- Snyk
- Checkmarx
The future of DevSecOps.
Data is being created as a byproduct of the new tooling and focus around security. This data is what we will see change the future of DevSecOps. The amount of data being created by logging tools, configuration management, deployments, and other sources is going to require analysis that may go beyond simple metrics.
Artificial Intelligence (AI) and Machine Learning (ML) are going to be vital in order to make sense of everything that’s being monitored in various environments. This inclusion of AI into current practices may mean less and less of a need for operations. Imagine software and its infrastructure becoming so acutely aware of scaling needs, it does so ahead of time. Or a system that learns the difference between an inconsequential attack vs. one that requires human interaction.
DevSecOps is one of the hottest aspects of DevOps today. In the recent Yalla DevOps 2021 event, for example, discussion around DevSecOps was very high on the agenda, as summarized in one of our recent posts.
This and more is what we have to look forward to in DevOps Security. Done well, it is a discipline that promises to save hours of work and possible headaches in the future. Beyond the “CYA” aspect, it is a great example of how DevOps was once a silo but is now a mindset for everyone involved. The additional collaboration and checks along the development process are sure to surface other opportunities to promote better security in our daily work.
