Getting A Good Metrics Program Off The Ground

To see why a good metrics program is essential, Andrew Jaquith, chief technology officer of Perimeter e-Security, points to a recent meeting with a major -- and, of course, unnamed -- client.

This particular customer spends many millions of dollars on Perimeter's e-mail services, Jaquith told attendees at last week's RSA Conference in San Francisco. Without good metrics to show the value contributed by the security service provider, it's unlikely they would remain a client.

"The challenge we have, and our sponsor was quite plain with us, he said, 'Look, our CFO signs a very large check to you every year. You need to show me that the value of what we are buying is justifying the investment,'" Jaquith said. "It's not that he wants to see a a $2 million security return based on cleaner e-mail ... but he does want to see that, based on what we are doing, that it's making a difference to his business."

A good metrics program is all about showing the value of certain security choices to decision makers. Obviously, it's not just about customers. Security teams also have to satisfy their "internal customers" -- the executives who are signing off on budgets, as well as educating the rank-and-file to take information security to heart. "The thing to remember is that metrics need to be motivating," Jaquith said.

There are three types of metrics: tactical metrics that must be acted on in near real-time; tactical metrics that don't need to be acted on in real time; and strategic metrics, which should directly impact the company's bottom line, said Alex Hutton, a member of the RSA panel and director of operational risk for an unnamed financial institution.

"The top strategic metric, of course, is that almighty dollar," Hutton said. "If you can't express your strategic metric in terms of dollars, you are going to run into problems."

Here are four recommendations from the panelists at the RSA Conference.

1. Start collecting data -- now.
When thinking about starting a metrics program or expanding an existing one, many security professionals will overthink the problem and run into analysis paralysis, said Arian Evans, vice president of operations for Web security firm White Hat Security. "Security metrics are a lot like diet plans. Most people I work with spend more time preparing to diet or collect metrics than actually going out and doing it."

Most business owners want to use metrics to make a qualitative decisions, he says. They think they need to have great data to do that, but determining what data is good is difficult until a company is already collecting information and trying to analyze it. So start collecting now, he said.

"You don't know what you are going to find, so start small and keep it simple," he said.

2. Develop and refine your models.
When asked is they liked their metrics programs, very few attendees raised their hands. The reason for that is metrics are only as good as the questions that you are asking of them, Hutton said.

"Your dissatisfaction with your data has nothing to do with your metrics program, but with your models," he told attendees.

Instead, companies should take a look at their standards and assumptions they use in analyzing their data. Most likely they will find that their models are not very good. Until they know more about the questions they are asking and why they are seeking them in that particular data set, the answers are meaningless.

Rather than trying to collect all of the data and look for patterns, companies should make hypotheses and collect data that will prove or disprove those hypotheses.

"Ask a simple question and try to answer that question," said White Hat's Evans. "Come up with three metrics, pick three metrics, and you will probably get it wrong ... but it is all about measuring and trying to figure out what you are missing."

3. Go beyond benchmarking.
While benchmarking your company against others in your vertical industry can allow businesses to have some idea of whether they are, say, spending a similar amount on security as the rest of the industry, driving your corporate security program based on others -- even those companies in the same industry -- is not a good use of metrics, Hutton said.

"If you management wants to compare you to someone else, you have weak management," he said. "It's all about risk tolerance. There are limits to that type of analysis."

Yet both Evans and Jaquith argued that knowing your ranking can be good, especially if it shows that a company is spending way too little money on security. Such a revelation can drive management, especially if it can be argued that a company on the bottom rungs of security spending is not doing its due diligence.

4. "Presentation matters."
Finally, developing good metrics does not matter unless the data is presented correctly and put in context for the target audience. Part of that is comparing and contrasting to previous performance data.

"Presentation matters and narrative matters," Perimeter's Jaquith said. "There is always some data that explains why something is happening."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.