Moving DevOps Security Out of 'the Stone Age'

Combining software development, deployment, and operations pipelines into DevOps teams promises increased efficiency, easier and more frequent updates, and better quality applications. Yet the complexity of the infrastructure has also led to a growing attack surface that is hard to monitor and maintain.

On the development side, the average organization uses four to nine different programming languages, has to deal with millions of new packages and images every year, and remediates thousands of vulnerabilities in the most common open source components, according to JFrog's Software Supply Chain State of the Union 2024 report. At the other end of the DevOps pipeline, two-thirds of companies have delayed deployment of an application due to Kubernetes security concerns, and nearly half (46%) had actual security incidents, according to Red Hat's The State of Kubernetes Security 2024 report.

Cybersecurity professionals aiming to secure the application pipeline have to pay attention to the software being written by developers, the open source components imported by developers, the containers and cloud infrastructure used to deploy software, and the build tools used to make the software, says Jeff Williams, chief technology officer and co-founder of Contrast Security, a software security firm.

"The problem is it's such a huge attack surface," he says. "It's not just your pipeline. It's all the other code that goes into developing software — it's IDEs and test tools and performance suites ... any one of them is capable of subverting the code that your developers are building and producing."

Gaining an integrated view of the entire DevOps pipeline, from development to application deployment, is increasingly important. Software components — not just open source libraries but Docker containers and other infrastructure assets — often have vulnerable code, increasing risk. Third-party tools can be compromised — remember Codecov's breach —allowing malicious code to be injected into projects under development. Cloud infrastructure and storage can be misconfigured or improperly protected, a la Snowflake instances.

Having good visibility into the state of the DevOps software pipeline and deployment infrastructure is critical, says Josh Lemos, chief information security officer at DevOps provider GitLab (and no relation to the author).

"There are two real important trains that need to run," he says. "One is you need the development and packaging security, compliance, and attestation of all of your build artifacts in one of those trains — or work streams. The other is the deployment monitoring and orchestration of those things in your production environments."

Write, Use, Buy, Build

Overall, DevOps security teams need to protect four areas that are open to attack. The first and second areas are most obvious to developers: the code that they write and the software components that they use, says Contrast Security's Williams.

"We've been talking about [that code] since the beginning of OWASP ... if you got bugs in the code you write, people exploit them, and you get breached. It's not good," he says.

Companies also have to pay attention to the code that they buy or — through a service — use indirectly. Finally, they need to secure the applications and services that are used to build and deploy software — the IDEs, test tools, performance suites, and instrumentation.

"Any one of those is capable of subverting the final code," Williams says, adding that most DevOps teams do not pay attention to the full attack surface posed by their pipeline and software supply chain. "I think we're still in the Stone Age, when it comes to real supply chain security."

While the vast majority of companies (87%) are building or moving applications to cloud-native, most (59%) did not understand the security implications of doing so and have suffered a security issue as a result. Predictably, the collection of common security incidents are as varied as the infrastructure needed to produce and deploy software: Network breaches, API vulnerabilities, certificate misconfigurations, cluster misconfigurations, and vulnerabilities in containers are among the top causes of security incidents, according to a November 2023 survey of cloud-native application security issues.

Even companies that are monitoring parts of their DevOps pipelines are not getting good coverage, says Williams.

"It's not everywhere, and almost nothing covers part of the DevOps like developer workstations and IDEs and testing frameworks and plugins," he says. "I mean, there's a universe of code that nobody's monitoring, and most organizations are not really thinking about this problem."

Questioning Your DevOps Infrastructure

For most companies, ensuring that they have visibility into the entire pipeline is essential. Monitoring can warn when a retired package is suddenly revived in the repository by an untrusted party, or when secrets are included in code that might otherwise be pushed to a repository, or when a Docker image has significant amounts of unused software.

Companies need to have continuous monitoring of each step in the pipeline, says Paul Davis, field CISO at software supply chain provider JFrog.

"[Knowing] what's going ... and [seeing that] a package has gone bad in production, or that I need to roll back a package because somebody's come with a new vulnerability — that ease of use [and visibility] into the attack surface, that insight and that traceability — is key for me," he says.

Companies should also take action around four specific areas of their DevOps infrastructure, according to GitLab's Lemos. First, the identities of any developer, ops specialist, device or service that takes part in the pipeline should be logged. Companies should also maintain a list of software artifact that they are using, which ones have vulnerabilities, and maintain a private repository, if possible. The build systems should be frequently tested, and any automated triggers — such as changes to third-party software that triggers a build — should be analyzed for potential security implications. Finally, the entire pipeline should be architected to minimize the impact — the "blast radius" — of a compromise, he says.

"The best thing I've seen companies do as a first step is to get to some known good design patterns," Lemos says. "The more of that that you can abstract away from [bad security practices], the more successful your security program will be, the less churn and load you'll have, and the more reusable your code becomes."

The Promise, and Peril, of AI

The breadth of the DevOps attack surface also represents an opportunity for automation and AI assistance. DevOps already gains much of its agility and speed through automation, with configuration- and infrastructure-as-code dominating because expressing architecture as files allow repeatability for operations, while analyzing the instructions allow for more secure infrastructure.

Yet, when it comes to security, most companies are holding back on adoption, says Laurent Gil, chief product officer for Kubernetes automation platform Cast AI.

"Almost every security company offers automation in some form, and yet nobody is using it," he says. "[Security teams] should know that it's okay to use automation to either block things that should be blocked, or to auto-remediate when you find something that contains vulnerabilities."

AI development also brings new ways of working with code and data — an attack surface area that is not fully understood and for which DevOps teams are not ready, says GitLab's Lemos.

"There is the possibility to do really old-style attacks, because you're combining data and content into a model," he says. "A model with a pickle file that gets consumed into a data scientist's workstation, if they deserialize it and it has a payload, they've just invited in some malicious code into their environment."