To Secure DevOps, Security Teams Must be Agile

A lack of resources, disruption from the pandemic, and a failure to integrate security into the DevOps pipeline have left many companies struggling to secure their applications — and security teams trying to catch up with the pace of development, said experts at the SecTor security conference this week.

While 83% of chief information security officers (CISOs) see software vulnerabilities as a threat to their organizations, nearly two-thirds of security teams are playing catch-up with the modern software development life cycle (SDLC) and falling behind, said Will Kapcio, a solutions engineer for HackerOne, during a presentation about DevOps security. 

The disruption to business has exacerbated the problems, with 30% of companies switching resources from security apps to securing remote workers and another third seeing their security teams reduced.

The winnowing of security resources and its impact on business innovation are worrying CISOs, Kapcio said.

"We know there are vulnerabilities in our online services. We know that our technology value streams as they speed up are introducing vulnerabilities at an increasing rate," he said. "In the worst case, we are slowing down the flow rather than removing obstacles in adapting to a modern SDLC because we are worried about introducing new vulnerabilities and increasing our risk."

Agile development and DevOps have become a key way forward for many companies that are trying to innovate with software and services, but security has struggled to keep up. Since the release of the Agile Manifesto in 2001, application development has evolved from waterfall-style development to agile development, to agile infrastructure, and to continuous integration and continuous delivery (CI/CD).

Yet many aspects of the development process remain manual, which shuts out security from gaining visibility into the security of any particular application and prevents collaboration with the DevOps teams, said Yoni Leitersdorf, CEO and founder of Indeni Cloudrail, during a presentation at the SecTor conference.

Most companies use tools to analyze their cloud environments for misconfigurations and vulnerabilities, but those tools often do not fit well into an agile development process.

"It is not very actionable because as a security practitioner, you cannot make any changes to the cloud environment," he said. "And if you go to the infrastructure team and say, 'Hey, guys, we found all these issues in the cloud environment, let's fix them,' they will tell you to open tickets and prioritize ... and most issues they don't get to."

Three Pillars of DevOps
Part of CI/CD is the push to make every part of development managed by configuration files that developers and operations teams can modify and push live. Infrastructure-as-code and security-as-code are both part of this evolution. Yet to continue to improve, companies must embrace three pillars of DevOps: the flow of code from multiple minds to production, using feedback to guide DevOps teams down the right path, and learning continuously. That includes integrating lessons into automated systems to avoid future mistakes, Kapcio said.

Many software development and security teams have not embraced those lessons, he said.

"Security disrupts flow, provides negative feedback, and never seems to learn," Kapcio said. "We have new bugs all the time, and this rate is only increasing with more organizations moving to implement agile and DevOps. If security issues are caught earlier in the life cycle, they take less time to fix, and that is where a bug-bounty program can help."

HackerOne uses DevOps in its own processes, pushing code around 10 times a day to production and releasing three to six new features every month, Kapcio said. The company tracks a variety of metrics, including cycle time, throughput per developer, change failure rate, and mean time to resolution.

Kapcio argued that bug bounties increase agility, which is not surprising considering HackerOne is a provider of bug-bounty management services. Hackers and bug bounties are about finding vulnerabilities, fixing those security issues, and using that feedback to inform application development, he said. In more than three-quarters of bug-bounty programs — 77% — hackers find a valid vulnerability in the first 24 hours.

Yet Indeni Cloudrail's Leitersdorf pushed for integrating security into the same processes that developers are using for functional testing and code checking. By using the same processes, security rides along with developers, rather than attempting to direct their teams, he said.

"The same concepts that are being used for functional testing of application code can be used for security testing of infrastructure," Leitersdorf said. "And that is something that engineering leaders are getting behind because it fits what they are already doing with application deployment."

Focusing on a pipeline using infrastructure-as-code allows security teams to build in static analysis tools to catch vulnerabilities early, dynamic analysis tools to catch issues in staging and production, and policy enforcement tools to continuously validate that the infrastructure is compliant, Leitersdorf said.

"If you think about how security can be done now, instead of doing security at the tail end of the process ... you can now do security from the beginning through every step in the process all the way to the end. Most security issues will be caught very early on, and then a handful of them will be caught in the live environment and then remediated very quickly," he said.

Developers get to retain their speed of development and deployment of applications and, at the same time, reduce the time to remediate security issues. And security teams get to collaborate more closely with DevOps teams, he said.

"From a security team perspective, you feel better, you feel more confident, you have guardrails around your developers to reduce the chance of making mistakes along the way and building insecure infrastructure and you now have visibility into their DevOps process, a huge bonus," Leitersdorf said. "This is the future — the future is infrastructure-as-code security and doing cloud security in a way that developers can understand and interact with."