How To Put Data At The Heart Of Your Security Practice

“When you can measure what you are speaking about, and express it in numbers, you know something about it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts advanced to the stage of science.” -- Lord Kelvin, 1883

Lord Kelvin wrote those words over 130 years ago. He was addressing a group of civil engineers on the topic of practical applications of electricity. While electricity as a physical science may (or may not) seem like a far jump from information security, how we improve our knowledge and understanding has remained relatively constant: we learn by observing and measuring our environment. Security programs are just now beginning to realize this lesson and there are a lot of questions being asked around data-driven security programs and how to build a security practice that has data analysis at the heart of the decision-making process. 

However, before we talk about how to approach the integration of data and measurement into the security decision-making process, let’s talk about how you should not to begin. This is important because many data-driven security programs are doomed in their first step because people don’t ask the right questions. Instead, organizations will look at the data they have and try to pull out things that are “interesting” or that they think will help drive their program. This only leads to metrics that are convenient and not necessarily useful and will end up wasting a lot of time and energy from everyone involved.

Anything worth doing is worth asking questions about
To build a data-driven security practice, start by defining a list of questions that, if answered, would help not only drive decisions, but also help how you evaluate how good those decisions were down the road. Defining such questions are tricky, because they can’t be just any old questions, they must seek out objective answers. As Bill James, who spent his life studying and reporting on the statistics of baseball, once said, “My job was to find questions about baseball that have objective answers, that’s all that I do, that’s all that I’ve done.” So rather than ask, “How secure am I?” perhaps a better question is “How many security events did we have last quarter?” Or maybe even dig deeper with, “What types of security events do we spend the most time on?” Through this approach, you can objectively answer, “How secure am I?” with multiple points that are grounded in data.

Another approach is to pause before you sign off on the next security purchase and ask what observable actions in the environment the team would expect to influence with this purchase. This is not an easy challenge; sometimes decisions are made to stop something that hasn’t happened yet. In that case, the questions may look externally, “How many breaches were disclosed by other organizations like us?” Taking an outside-in approach will broaden the sources of data and help answer some those tough questions. 

Once you have a list of questions that you’d like answered, look for data sources to answer them.  Chances are extremely high that you aren’t collecting all the data you’ll need. The good news is that many organizations are asking the same questions and vendors are beginning to respond with data-driven solutions. For example, perhaps you don’t have to measure all of your industry peers: there are vendors and industry reports offering up answers that you can draw from.

Simple answers are still answers
Start trying to answer your questions with simple counts. Counting things is the first big step in being data-driven and the number of questions that can be answered with a simple count may surprise you. Simple counts paint with large brush strokes and may answer many of your initial questions. But sooner or later, you will want to compare two different counts. Perhaps the comparison will be as simple as comparing one month to the next, but eventually the need to compare two different counts is inevitable. Be forewarned: this comparison is the second big step towards being data-driven. Someone will ask, “Is the difference significant?” and that new question will set you traveling down a path towards statistical thinking. Don’t panic, statistics have already helped shape the evolution of many other fields and resistance is futile.

There is good news and bad news at this point. The good news is that we can get a lot of answers (and therefore benefit) with some relatively entry-level calculations.

The bad news is that the list of questions will grow exponentially as previous ones get answered. “How do I compare to my peers?” and “How can I measure my vendors and third parties?” Congratulations, you are well on your way with a security practice that has data analysis at the heart of the decision-making process!