Searching For Security's Yardstick

There’s an old saying in IT: You can’t manage what you can’t measure. If that’s true, however, security managers must be in a world of hurt.

Across this usually contentious security industry, there is violent agreement about two points: Security departments need better ways to prove that their organizations are safe, and there are no clear-cut numbers that definitively prove that point.

"So you’re in the management meeting, and the sales guy gives specific numbers about orders and gross revenue," says Steve Dauber, vice president of marketing at RedSeal, which makes software designed to monitor security posture. "The networking guy gives numbers about uptime and throughput and response time. Then it comes around to the security guy, and he says, 'Well, we didn’t get hacked today.'"

The basic problem, experts say, is that it’s tough to measure a negative. If security’s primary goal is to prevent outsiders from getting in -- and insider data from getting out -- what numbers are there to measure its success? The only clear metric is a negative: How many times has a compromise been discovered?

"The measure of success in security is that nothing bad happened," says Mike Rothman, an analyst for Securosis, a security consulting firm. "Your best day is going to be that zero bad things occurred. There's never going to be a measurement that shows that good things are happening."

If security is about prevention of leaks and attacks, then what metrics should security departments show their bosses to prove that they are doing their jobs well?

"I think you have to start with things you can control," says Scott Crawford, an analyst at Enterprise Management Associates, a consulting firm that focuses on systems and network management. "If you can't change the controls, then metrics won't do you any good."

Setting a security policy -- and the means to monitor it -- is a good place to start, Crawford says. "If you set a policy and there is a growing number of systems or users that are operating outside the policy, then that's something you can act on, either through education or through greater controls," he observes.

But security professionals should be wary of "dashboards" and artificial measures that don't have meaning for the specific business that their enterprises are in, says Gary Hinson, CEO of security consulting firm iSecT in New Zealand.

"Some companies begin with a long list of 'security things that can be measured' and then try to shoehorn them into some sort of metrics system or dashboard. That, to me, is the wrong way to go about things," Hinson says. "You don't design an aircraft cockpit's information systems with a list of things that can be readily measured on the aircraft. You start by asking what does the pilot need to know -- altitude, azimuth/heading, etc. -- and then prioritizing those things, organizing them into related groupings, and finally filling the dashboard.

"Then you get lots and lots of feedback from pilots about what is missing, superfluous, misleading, wrongly positioned, too big/too small, too annoying/too discreet, etc.," Hinson continues. "In other words, the metrics design process is very interactive, involving the system designers, instrumentation specialists, engineers, and pilots all working together to define, design, and refine the metrics system."

Next: Basic metrics But no matter how customized your measurement system, every company needs some basic metrics to start off with, notes Steven Piliero, who heads up the Benchmarks Division of the Center for Internet Security (CIS). The CIS Consensus Information Security Metrics benchmark is perhaps the closest thing the industry has to a set of standards for security metrics today.

"There are three kinds of metrics: those that are broad enough and well-enough understood that they can be used across industries; those that are industry-sector specific; and those that are organization-specific," Piliero says. "We're helping to define that first category, the metrics that many industries can use."

The CIS Consensus defines some basic metrics that organizations can measure frequently, as companies do with certain financial numbers, or as hospitals measure post-surgical infection rates, Piliero says. "They're a starting point for building out your metrics -- some unambiguous standards for measuring specific security functions."

"It is possible to get some level of agreement on high-level metrics," Rothman agrees. "CIS Consensus is a great resource to kick-start the metrics effort."

The CIS Consensus offers standardized methods for tracking measurable activities, such as the frequency of incidents and the time/cost to mitigate them; scanning of vulnerabilities and the time/cost to repair them; and the frequency/time required to do patch management.

"You can measure things like the number of times you investigated potential indicators of anomalous activity," says EMA's Crawford. "You can track the number of cases of investigation and the number of cases that have had to be escalated to mitigation. You can measure the percent of unplanned IT work related to that escalation, and the resulting security spend."

However, experts warn against measuring aspects of security that may not be meaningful to the business -- or worse, may cause the security department to focus its efforts on the wrong priorities.

"Tracking vulnerabilities, days to patch, [antivirus] performance -- these might be useful at an operational level, but measuring these in order to show security effectiveness is a load of crap," Rothman says.

And while tracking incident response or mitigation time might be useful in benchmarking the performance of the security department for upper management, these metrics still don't provide enough meat to serve as a gauge for the organization's security posture, experts advise.

"To measure security posture means arriving at compound and composite metrics -- something like the cost of sales numbers that many companies track," says CIS' Piliero. "I'm not aware of any standard metrics that can measure that today."

There is an emerging class of tools for security posture management (SPOM), such as those made by RedSeal, currently on the market. Such products harvest firewall configuration data and other information to show the potential for access to critical business data -- a measure of both vulnerabilities and risk.

"Companies use us to do a risk analysis on a specific vulnerability -- what's the potential impact if it's exploited?" Dauber explains. "They can use this data to help with prioritization of security actions -- to help figure out what issues they should handle first."

Rothman says the SPOM concept has merit for measuring security posture, but the market hasn't taken off. "The big problem is the cost," he says. "Executives have to see that it's worth that much to be able to judge security posture, and that's only going to happen in industries where that sort of data is critical to the business."

Still, there is clearly a need for tools that can not only provide simple metrics for reporting to upper management, but can provide real insight into the company's state of security, Rothman observes. Core Security's new Core Insight tool -- essentially a penetration-testing appliance -- is one such emerging product, and nCircle's Suite 360 Intelligence Hub offers a way to benchmark one company's security against other, similar companies, he notes.

"One promising way to get some security metrics is to benchmark one organization's state and processes relative to others," Rothman says. "The problem with that is how do you attribute the data back without giving away too much about its source? Sharing between security companies is still the main constraint on this."

Organizations such as the CIS and the new Open Security Intelligence forum are attempting to provide a basis for the definition and sharing of security data and metrics, but there is still a lot of work to be done, experts say. Part of the problem is that there are so many different functions and players in the security metrics game.

"There is still a big gap between operations people, compliance people, and security people," says Joe Gottlieb, CEO of SenSage and founder of the Open Security Intelligence initiative. SenSage earlier this week published a study in which a majority of security people said they thought their security processes are less effective because data is not effectively shared among the various functional areas, such as compliance, incident response, and real-time monitoring.

"Most security metrics [initiatives] start because some enlightened executive up the chain asks for the numbers," says CIS' Piliero. "Once that happens, you see companies trying to get their own house in order, working together to pull together operational metrics before they start reporting up the chain."

Have a comment on this story? Please click "Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.