Vulnerability Severity Scores Make For Poor Patching Priority, Researchers Find

Relying on the measure of vulnerability severity to prioritize what to patch and what to put off for another day is a waste of effort on software flaws that pose no danger, while missing others that are being exploited, according to two researchers that plan to reveal their findings at the Black Hat Security Briefings later this year.

The research analyzed the severity of vulnerabilities as ranked on the popular Common Vulnerability Scoring System (CVSS) with the existence of exploits and whether those exploits were being used in the wild to attack systems. The researchers -- Luca Allodi, a Ph.D. student in the field of security economics at the University of Trento in Italy, and Fabio Massacci, professor of information systems and security at UT -- found that the CVSS score did not correlate strongly with the attribute that arguably matters most to companies: whether the vulnerability is being used to attack systems.

"The CVSS could be high, but you may have a low risk of being exploited, while you can get a low CVSS score and still be attacked," Massacci says. "There is not much correlation between the CVSS only and the chance of being attacked."

The Common Vulnerability Scoring System uses a number of qualitative characteristics of a software flaw to determine the severity of the vulnerability on a 10-point scale. The CVSS combines a number of metrics -- such as the complexity of the attack and whether it impacts a system's confidentiality, integrity, and availability -- to come up with the score.

The researchers compared CVSS scores from the National Vulnerability Database (NVD) with information from the Exploit Database on the subset of vulnerabilities for which exploits had been created and with information from Symantec on the vulnerabilities that were actually being targeted by attackers in the wild.

Vulnerabilities targeted by exploits for sale in the underground should be patched immediately, as there was a strong correlation between the sale of an exploit for a particular vulnerability and the danger of that vulnerability being attacked. However, there was less correlation between the existence of a proof-of-concept attack in the Exploit Database and the risk of attack.

The complexity of the attack -- one of the metrics used to make up the CVSS score -- also appears to have a stronger correlation to the chance of a vulnerability being targeted by attackers than the overall score itself, the researchers say.

"If your vulnerability is in an exploit kit, then patch," Allodi says. "And if it is easy to exploit, then patch. But if it is difficult -- more complex -- to exploit, then it depends on the importance of the software with a vulnerability."

[With flaw tallies varying by up to 75 percent, vulnerability data needs to be taken with a grain of salt, yet reports based on the data fail to include caveats, Black Hat presenters say. See Don't Take Vulnerability Counts At Face Value.]

Many of the criticisms echo those of researcher Dan Guido, co-founder and CEO of security startup Trail of Bits, who argued that companies should focus on which vulnerabilities are being attacked and find simple defenses that defeat the attacks. In a 2011 study of vulnerabilities targeted by popular exploit kits, for example, Guido found two mitigations that could block 90 percent of the attacks.

Doing that sort of analysis with CVSS scores is impossible, he says. The scores do not provide enough information to the information security managers, especially because two aspects of an attack are only known by the potential victim.

"The vendor has no idea what the company's network looks like and what the attacker might be after," Guido says. "And without those two critical pieces of information, it's hard to make the CVSS score relevant."

While the research highlights that CVSS has weaknesses, the scoring system is a good standard by which companies can express a single severity for software flaws, says Wolfgang Kandek, chief technology officer of vulnerability management firm Qualys. While Qualys does not use CVSS as the measure of severity for software flaws in its own service, the framework is good for the majority of companies, he says.

"It depends on the level of sophistication," Kandek says. "Our customers are good with our severity, and I know that some very sophisticated customers can pull apart CVSS values to make their own decision, but for most companies the straight score is a good measure."

Yet for companies who are trying to find the best use of their resources, focusing on CVSS scores to prioritize patching will waste effort, argues UT's Massacci. In many ways, prioritizing patching based on CVSS scores is like triaging patients in an emergency room by just their temperature, he says.

"A single number is not a good idea -- CVSS is like measuring you for a temperature and then sending you to the operating room if it's high," he says. "What you should do, like in the medical domain, is first measure if you have a fever, and then you do a blood test, and then you do an X-ray."

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.