Secure Code Starts With Measuring What Developers Know

Since 1999, I’ve taught over 2,000 developers, architects, and managers about application security. This is no small challenge, since the subject is almost totally ignored in most college curriculums and there is a lot to learn. In fact, the MITRE CWE Project lists over 1,000 different ­categories of security mistakes that developers can make. Many of these security quagmires are not immediately obvious and quite a few are downright diabolical. So I totally understand why developers don’t spend their off-hours researching the inner workings of "padding oracle" vulnerabilities and other security lore.

Still, we need developers to avoid making security mistakes that endanger their company and their users alike. Instructor-led training and e-learning are surprisingly effective and critical parts of an application security program. In one very large organization, we found that projects where more than half the team members had received secure coding training, the number of vulnerabilities plummeted by 73 percent. That result is far superior to anything penetration testing programs or automated tools could hope to achieve.

Despite many successes, I recently discovered that I’ve been teaching blindly. I have simply assumed what I thought my students needed to learn. We realized that measuring what students know, both before and after teaching, could help us provide more effective instruction. So we created “Secure Coder Analytics,” a measurement platform that analyzes a development team’s security knowledge. To ensure that developers don’t feel pressured, the tool protects participants’ anonymity.

Secure Coder Analytics draws questions from a pool of 500 questions that cover over 60 different secure coding subject areas. We have vetted the questions and answers for two years with real software development teams. Both the questions selected and the answers to those questions are fully randomized. While the questions are not easy, they have proven to be a reliable evaluation of a developer’s knowledge and skill in each area. Over 1,000 developers from around the world have participated, and the aggregate results are revealing.

This chart above shows the five weakest and five strongest areas. The weakest are at or just above the "random guess" level, and are a real cause for concern. However, I am encouraged by the strong scores in important areas like "Preventing Forged Requests" and "Protecting Credentials."

The results for your organization will almost certainly be different. We’ve found that some organizations do quite well in areas that others totally fail. However, the results are fairly consistent within a particular organization. This suggests that different organizations are successfully teaching their developers about certain security areas. We hope to increase visibility and expand this training to cover what’s really important with Secure Coder Analytics, which I encourage you to try out for yourself. In the meantime, let’s chat about what you think your developers do and don’t know about application security in the comments.