Using DevOps To Upgrade Application Security

SAN FRANCISCO -- RSA CONFERENCE 2013 -- Traditionally, the world of IT app development and deployment has pitted developers and IT operations staff in two corners of the proverbial ring, making them what some experts call "natural enemies," with little cooperation and a lot of bureaucratic processing in the march toward pushing production code live. It's an unproductive cycle that has spurred the countercultural DevOps movement.

The idea behind DevOps methodology is to eliminate clashes between the two groups and implement faster, more bite-size code deploys at more frequent intervals. The volume and speed of deploy rates, plus the shift in engineering culture precipitated by DevOps, could scare some security pros. But a panel at the RSA Conference last week showed how the paradigm shift could actually provide a huge step forward for security teams seeking to insert themselves into the development process for a more rugged enterprise application infrastructure.

"What I find so amazing about studying DevOps organizations is that they have a culture that embraces security," said Nick Galbreath, vice president of engineering for IPONWEB. He described DevOps as a much more healthy way of developing code within an enterprise. "The great thing is that all of the tools that you use to enable security layers right on top of DevOps. Having these tools that developers in operations use together to make things to go faster is just a great way for you to get your [security] job done."

According to Gene Kim, founder and former CEO of Tripwire and author of "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win," DevOps breaks the "core point of conflict" between developers and operations staff.

"What you get is phenomenal feature deploy rates as well as world-class stability and security," he said, pointing to organizations like Facebook and Netflix that can see as many as 23,000 deployments in a day, but stressing that organizations don't necessarily have to work at that pace to take advantage of DevOps.

One of the big security worries that crops up with such a deployment schedule is the pressure it puts on change management, code review, and other aspects of security scrutiny, he said.

"Any place that you have manual review, you just can't do that 1,000 times a day," Kim said. "Those manual reviews have to be turned into some sort of automated tests that get put into the continuous deployment pipeline."

However, this dovetails nicely into one of the four fundamentals that panelist David Mortman, chief security architect for enStratus, identified as critical to implementing DevOps methodology: culture, automation, measurement, and sharing.

"For me, DevOps is about doing your job right," he said. "There's a lot about speed and things like that, but for me it's about working effectively with other team members."

Automation plays a huge part, and Mortman said that security can easily piggyback on the automated testing that DevOps naturally encourages.

"If you watch a lot of DevOps talks on the dev side, they talk about automating all of your unit tests and functional tests and integration tests," he said. "So one of the things I'm doing is working with one of our engineers to add security unit tests and functional tests to the code they're already writing. So that way, every time someone gets code properly committed, it gets tested for all of these things [and] if someone broke something or potentially broke something ... you find out immediately."

It's a principal that Twitter uses in its stack code analysis, Kim told the crowd.

"Basically, every time a developer hits 'save,' it runs static code analysis, and they'll get an email that said you just wrote a piece of code that creates this vulnerability, and here's how you fix it," he said. "It's not on code commit -- it's on save. That is integrating security in the daily work of developers and operations."

DevOps also affords organizations more opportunities to fix issues quickly, said Josh Corman, director of security intelligence for Akamai Technologies.

"The ability to do smaller batch sizes and more of them with more confidence makes it so you don't wait six months to get your security patch in," he said. "You're no longer afraid of patching because you can always unpatch or roll back."

More importantly, the faster speed of deployment and the automation makes it easier to build in a sense of immediacy with the security feedback offered to developers, greatly increasing the opportunity for not just fast fixes, but also better developer education. According to Galbreath, the traditional development cycle can have the security team pointing out problems to a developer who has long since moved past the relevant project.

"Now they have to go find the developer, who honestly doesn't remember anything about what happened," he said. "It's just an annoyance. The fast deploy cycles really change things: You go and say, 'That deploy you did yesterday, there's a problem with it,' and they say, 'OK, let me get in the queue and I'll fix that later today.'"

The other big plus of frequent but smaller code deployments is the subsequent reduction of complexity within each deployment, Mortman said, who explains that studies have shown that increasing code complexity can have an exponential affect on security and stability.

"When you have less code, you have less complexity and you have fewer security bugs," he said.

According to Corman, organizations don't even necessarily need to transform their shops with DevOps methods -- even if legacy deployments get in the way of across the board changes. "I'm not saying you can wave a magic wand and fix all of the legacy stuff, but you don't have to make new stuff as badly as we made the old stuff," he said. "Even if you're not going to do DevOps, the lessons we've learned are equally applicable to our other stuff. In some of these cases, the way a security person got a seat at the table is by introducing DevOps to their development group."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.