Effort Will Measure Costs Of Monitoring, Managing Network Security

A new initiative launched this week will provide metrics to calculate the cost to organizations of monitoring and managing network devices and how those security operations are working -- or not.

Network Security Operations Quant, created by consulting firm Securosis this week and sponsored in part by SecureWorks, is a metrics project aimed at giving organizations a way to accurately measure how much these operations cost them both financially and resource-wise.

Securosis previously developed a similar set of metrics for quantifying the cost and efficiency of an organization's security patching process, and recently launched a metrics effort for database security.

The network security metrics model will be released under a Creative Commons license and initially focuses on five network security processes: monitoring firewalls, monitoring IDSes/IPSes, monitoring server devices, managing firewalls, and managing IDSes/IPSes.

Mike Rothman, analyst and president of Securosis, says the idea of the open-source metrics is to help organizations optimize how they monitor and manage their network security devices, and to be able to compare the costs of different options for doing so.

The metrics could help a company determine whether, for example, it's efficiently deploying its resources: "A lot of organizations are struggling with what to do with themselves: What should they outsource? What should they be tracking?" Rothman says. "We believe this model is going to help organizations understand [things like] if they have resourced their managed devices correctly and if their people are being [deployed] correctly."

Rothman says the metrics model will also provide a detailed look at the monitoring and management processes of these devices, as well. "We're trying to document all of the possible steps," he says. "But not everyone is doing everything, and that doesn't mean you aren't [good] at what you do...Part of the art of this is figuring out what's important to a specific organization."

The quality of security metrics matters, says Thomas Ptacek, founder and principal with Matasano Security. Metrics that are relevant to a company's business and get the attention of upper management are the valuable ones, he says. "These are the ones you take to the board meeting and get head-nods. These are totally valuable. You can tell when they are working when they drive decisions."

But industry best-practices type metrics aren't as effective, Ptacek says. "The one-size-fits-all approach doesn't work at all," he says.

One especially useful measure that's typically missing from metrics is the element of time, says John Pescatore, vice president and distinguished analyst at Gartner. "It's harder to measure but starting to be done in SIEM and forensics products," Pescatore says. "When a security event got in, how quickly did we notice? How quickly did we react, and how quickly did we close it out?"

If an event took three months to detect, then that's obviously bad, he says. "And the next time maybe it took 30 seconds to notice" an event, which would demonstrate an improvement in security event monitoring, he says.

"For metrics to be useful they have to be measurable in a way that doesn't cost a lot to measure, or you could spend too much of your budget measuring [things]," Pescatore says.

Ideally, you should measure things that are tied to a business problem, rather than a security problem, he says.

Why should enterprises care about network security metrics? Securosis' Rothman says metrics are crucial ammunition for security professionals when senior management asks them how effective they are and whether their security controls are working. "Part of the Quant philosophy is to help define these answers in an open way," he says. "There are a lot of TCO and ROI models out there [for security], but most of them are from vendors, and they are all slanted in some shape or form."

Securosis is recruiting enterprises to serve as anonymous subjects that will provide the basis for its research in actual network security processes. Rothman says he hopes to have the mapping function finished in April. "We want to start posting the initial versions of the process framework in late April or early May. The goal is to have something workable and tangible to start collaborating among the community then," Securosis' Rothman says.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.