The Staggering Complexity of Application Security

First, a few facts.

A typical, midsized financial organization has a portfolio of over a thousand applications. The largest firms exceed ten thousand applications. Each of these applications, on average, has hundreds of thousands of lines of custom code, and the largest can have over ten million lines. In addition, each application also includes anywhere from dozens to hundreds of software libraries, frameworks, and components that typically total over ten times the size of the custom code. And this portfolio is growing rapidly -- over 20% of these applications have new and updated code each year.

By comparison, consider the US Federal Tax Code, which has also grown dramatically over the years. Currently, the tax code totals just 4.4 million lines of “code” – roughly equivalent to just a handful of applications. As a security researcher I’ve discovered thousands of vulnerabilities in code. But as a former CEO, I’ve also analyzed a ton of legal contracts for loopholes. What’s interesting is that whether I’m scrutinizing software code or reviewing legal language, the analysis is not as different as you might think. Both require a detailed understanding of specialized language and a solid understanding of the underlying business.

So after two decades of high speed coding, a typical large financial organization has a pile of code as large as 2,000 copies of the entire 73,954 pages in the US Federal Tax Code -- almost 10 billion lines of code.

The current security situation
After the details of JP Morgan’s breach were revealed recently, a former employee told The New York Times that it was as if “[the attackers] stole the schematics to the Capitol -- [JP Morgan] can’t just switch out every single door and window pane overnight.” Actually, I think it’s considerably worse than that. It took decades to create the software infrastructure at JP Morgan, and there is no practical way to just change it out. To me, it’s more like discovering that an entire city has been built with asbestos insulation, lead pipes, and unshielded power lines. And they all need to be swapped out without disrupting service.

Now, according to experts at Aspect Security, the typical enterprise web application has an eye-opening 22.4 serious vulnerabilities. They aren’t always easy to find, and they vary in likelihood and severity of exploit. But combining this level of vulnerability with an increasingly sophisticated threat has led to a marked increase in breaches. The breaches just this year have been sobering.

The limits of traditional AppSec
In the good old days, we did manual penetration testing and code review to find vulnerabilities. These reviews got pretty good coverage and developers had time to fix problems before the code went into production. But recent advances in software development, including widespread use of libraries and components, high-speed development methodologies, complex frameworks, and inscrutable protocols have made manual analysis too slow for all but the most critical applications.

Many industries reach a point when production speeds are maximized but quality can’t keep up. The automobile industry went through many difficult years retooling for increased quality. The Agile and DevOps communities have been successful at using faster iterations to keep software projects from straying too far afield. So we are building code faster and faster, but security hasn’t evolved. We simply have to find some new technologies and techniques for gaining assurance at DevOps speed and portfolio scale.

Refactoring AppSec for speed and scale
First, we need to abandon the notion of security exceptionalism, and learn from other industries. We can instrument the entire software development process -- design, development, test, and even production -- so that applications continuously test themselves and provide real time security feedback as they run. Essentially, we have to make security testing, intrusion detection and response, and runtime protection a part of every application.

Organizations like Etsy, Netflix, Twitter, and Yelp have already realized the problem and have started to deploy new security tools that work differently. These tools are not legacy tools that run at the end of the development process. These tools instrument the software development process, gathering security information in real time as applications are built, integrated, tested, and deployed. Most importantly, these tools, like continuous integration and continuous delivery tools, don’t disrupt the normal software delivery process. Security tools that interfere with or slow down software delivery don’t get used. Zane Lackey from Signal Sciences put it best, “development organizations view delays as damage, and route around them.”

How can we get there? It’s not as hard as you might think. You can start today by creating a script, test case, or tool that performs a single simple security test and distributing it across your application portfolio. Or use the free Contrast for Eclipse plugin that I wrote about last month. Much of the infrastructure you’ll need has probably already been created by Agile and DevOps teams.

We need to reimagine all of our security testing techniques so that they make sense in a continuous environment. We also need our security experts to become coaches and toolsmiths rather than the ones to chase down every vulnerability -- because that will never scale.