9 Questions For A Healthy Application Security Program

One of the mementos that I've carried on my migration from software development to information security is a piece written by Joel Spolsky, of Stack Overflow fame. Called the "Joel Test," it's a quick, back-of-the-envelope indicator for whether a development team is set up for success or failure. Pointedly succinct but uncompromising, it helps cut through bureaucratic hedging, and it's always in the back of my mind when I'm working with development teams.

However, I've noticed that it's difficult for many developers—and management—to achieve this intuitive level of clarity about having the proper pieces in place to build software securely. Security is a special sort of requirement and can't be treated like just another nonfunctional requirement; the design and testing approaches are very different. To help organizations and teams discuss whether they are prepared to build secure software, I've taken inspiration from the original Joel Test to pose nine key questions about application security.

1. Do you have an application inventory?
Any organization developing or deploying multiple applications needs a single point of reference for metadata about applications. Questions such as "who are the points of contact?" "what does it interact with?" "where is it deployed?" "how sensitive are the systems and data?" "what are the dependencies?" and dozens of others are going to be of interest to groups in the organization outside the immediate development team. Tracking down individual developers is not a practical solution.

2. Can you deploy environments in one step?
Security testing is messy. In the best cases, it only fills up logs and eats bandwidth, but in the worst cases (and where it's needed most), it has a tendency to regularly make entire environments unusable for any other purpose. Organizations able to create complete environments easily are far better prepared to perform adequate testing, not just security testing.

An additional outshoot of this level of automation is that it reduces vulnerabilities stemming from misconfigurations and deployment issues.

3. Do you do threat modeling?
Security thinking is a different type of thinking. Threat modeling is a powerful tool for making sure that security concerns get a seat at the table during architectural and early feature discussions. Later on, when smaller details are being worked out, the actual threat model document helps make those concerns concrete.

4. Do you have a Secure Development Lifecycle?
It's easy to make the SDL a catch-all for security practices or get bogged down in the tremendous amount of content out there about implementing one. The overarching point, however, is that security considerations and practices get fully considered in every step of the process and intentionally integrated into the day-to-day work of creating software. A team diligently executing an SDL is going to tick a lot of boxes on this list.

5. Is secure development part of developer training?
Breaking systems requires a different mindset than building them. There's nothing that says an individual can't do both, it's just that they're different skills, and practicing the latter doesn't necessarily hone the former. Even when writing my own code, I will walk away from a project for a day or so to make sure that I've sufficiently purged any preconceptions about how the code "should" work. Effective developer training not only equips developers with prescriptive "do's" and "don'ts," but provides examples of attacks and attacker tools to help them don the appropriate mindset.

6. Do you choose technologies and approaches that eliminate entire classes of vulnerabilities?
In my experience, this is one of the most powerful determiners of whether an application is "mostly secure" or "mostly insecure." Smart pointers, ASLR/DEP, managed languages, prepared statements, the controller pattern, and Content Security Policy are all excellent examples. The ability to systemically remove entire areas of concern at the design phase can have huge impacts during implementation, testing, and maintenance. Any details that can be offloaded from a developer's mind onto a useful and secure-by-default technology stack are potential bugs that never get written.

7. Do you use automated scanning and testing tools?
Security testing often requires testing exponential combinations of payloads and contexts, far too many to be part of manually developed testing suites. Automation is the only tractable approach, and sophisticated tools exist for both static and dynamic analysis of software for a huge range of issues. Attackers automate mercilessly to find vulnerabilities and exploit them at scale. Defenders that fail to automate will find that important processes won't be done often or well.

8. Do you conduct penetration tests?
Security tools and automation can uncover a wide range of technical flaws; I'm often amazed at the obscurity of conditions and paths they discover. However, tools don't tend to be good at providing a realistic adversary simulation. Goal-based penetration testing is useful to find logic flaws and chains of lower-severity issues that can result in significant compromise. Penetration tests also give an attacker's view of the application; a legacy PHP site or broken "forgot my password" functionality can easily negate excellent security practices on an otherwise shiny new application.

9. Does software security have an advocate at the executive level?
All engineering requires balancing competing concerns, and security is another one jockeying for resources. A security initiative unable to allocate and defend resources (money, people, time) is destined to be marginalized.

As with the original Joel Test, an organization might be able to get by with missing one or two of these questions. However, a mature organization is going to nail every single one as a matter of course. The common theme here is that "secure development" really needs to be thought of as simply "development." Creating artificial distinctions between the two moves resources and expertise away from the places it can do the most good. Conversely, frontloading security as requirements and using tools and training to keep security considerations close to mind can eliminate issues sooner and cheaper.