Governance Without Metrics Is Just Dogma

Without a solid security metrics program, organizations will struggle to institute risk management in meaningful ways and could be basing their security on false assumptions, an expert panel warned at the RSA conference last week.

"You know what you call governance without metrics? Dogma," said Alex Hutton, director of operations risk and governance at Zions National Bank. "You know what you call governance guided by metrics? Risk management."

As Hutton sees it, there's very little separating governance and risk management, but without metrics to feed into risk models, measure security performance, and relate security controls to exposure to loss, enterprises will have a difficult go at managing risk. According to fellow panelists, metrics bring the rigor of discipline and measured decision-making to the security industry.

"The metrics that we try to use and leverage and develop are intended to inform and turn assumptions into understanding, more than anything else," said Jack Jones, principal of CXOWARE Inc. "We have as an industry a bad habit of being a little bit superficial in our treatment of the problems we face. If we want to evolve, we have to [have] a little more critical thinking in our approach."

As they stand by themselves, risk models are essentially hypotheses. To ensure that an enterprise is truly operating under a valid hypothesis, some sort of feedback loop needs to be instituting to test against it, Hutton said.

"If you talk about probabilistic pursuits, so someone who is an actuary or quantum physicist or whatever they are, they'll talk to you about concepts like model fit, model updating, yadda yadda, yadda," he said. "You've got to ask, where's the feedback loop?"

Panelists addressed the resource limitations that many security and risk professionals offer as an excuse to forgo developing a metrics program. But metrics don't have to take a lot of extra resources, said David Mortman, chief security architect at enStratus. They can often be found from existing data used in creative ways.

"You have more data than you think you have, and you don't need as much as you think you need," he said. "Test. See what happens."

According to Hutton, security metrics programs can't be bought, anyway.

"You can't buy the metrics program or the risk management program off the shelf. It's not like you can just hire a CISSP and say, 'You're the metrics guy,'" he said. "In fact, one might argue that that's the exact opposite thing you should do. You should find some kid from the local university with a biostats or an econometrics program who is also a very creative individual and bring them in to help them out."

Panel moderator John Johnson, global security program manager for John Deere, agreed that creative staffing can make a big difference in the industry and in bringing metrics to the forefront of risk management.

"You need someone with analytical skills, creative skills. My background is nuclear physics," he said. "You just don't know where talent is going to come from."

With or without help from a motley crew of number crunchers, metrics don't have to be an all-or-nothing factor in risk management, said Jones, who wanted to dispel the notion that metrics only work if a department is making quantitative decisions across the entire practice.

"That's ridiculous," he said. "There are marvelous opportunities for quantifying things that will make a tremendous difference in your ability to be smart about how you do your work, but most of the decisions that cross your desk on a daily basis you're going to find [are made using] expertise you've acquired over years."

Caroline Wong, director of IT governance and risk products for Symantec and former security guru for eBay, agrees.

"The strength of your security program does not come from a product -- it doesn't come from a big four consulting company, and it does not come from a framework," she said. "Actually, it comes from your brain."

Wong related a story of one of her first forays into developing a metrics program at eBay years ago when the firm brought in a Big Four accounting firm to help.

"So they said to us, 'Tell us all the technologies you have and what kind of data you have,' and what they gave to us for somewhat of an absurd amount of money was an Excel spreadsheet with all of these data sources," she said, explaining that this unsatisfactory result had her and her team creating something from scratch.

Where they found their success was through conversations with security and operations staff on the ground. That started by asking product developers, from a security perspective, what worried them most on the website. Vulnerabilities came up as a big answer, so the team started developing a baseline and then, rather than pushing a goal down the product development team's throat, another conversation had security asking what a reasonable goal would be.

"So I think metrics is not about sifting through a mountain of data and trying to derive something meaningful," she said. "I think it starts with a conversation. I think it starts with a goal, engagement of stakeholders, and proper reporting."

Proper framing of the metrics themselves is also critical, Jones said.

"When we're focusing on metrics, the question I ask is, 'What are we measuring? What's the value proposition of those numbers in terms of loss exposure?'" he said. "Is it telling you something about your ability to manage loss exposure over time? I'm trying to characterize or frame metrics in these terms, because otherwise they're just numbers, and they're a waste of resources."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.