Recent Vista Metrics: Don't Be Fooled

Microsoft's security strategy director, Jeff Jones' recent report card bestowing high marks on the security of his employer's most recent operating system release has garnered plenty of ink. But what's it mean?The good is that the number of first-year Windows vulnerabilities is trending down. Jones even put his neck (way, way) out detailing how Vista's vulnerability counts are lower than most every other operating system on the market. So much for not tooting your own horn. The bad news is that there are still too many vulnerabilities being uncovered. And the truth is that straight-up vulnerability count comparisons between operating systems don't tell you much.

First, kudos to Microsoft for sticking to its Trustworthy Computing efforts. Those efforts kicked off in 2002, and the results are starting to show in a big way. In fact, Vista is the first operating system to go completely through Microsoft's secure development processes - and the number of vulnerabilities has been cut nearly in half.

That means the quality of Microsoft's operating system has improved. And that's great news for its customers. It does not mean you are inherently more secure using Vista. Even the fact, by Microsoft's count, that Vista garnered fewer vulnerabilities in the first 12 months of public availability than Red Hat rhel4ws, Ubuntu 6.06 LTS, and Apple Mac OS X 10.4 doesn't make it more secure by choosing Vista. And for the case I'm making here, I mean more secure in actual, real-world, use.

While the report, available here, is interesting, it's just a start at evaluating the overall security of the operating system. These vulnerability counts may be good for Microsoft's own efforts at quality improvement, but by simply knowing that Microsoft fixed 36 vulnerabilities in Vista, versus 65 for XP doesn't amount to much for the rest of us.

In his report, Jones' writes that fewer vulnerabilities "make it easier to manage risk." As well as this: "All other things being equal, fewer patches mean more time to spend on other security projects to reduce risk. Not necessarily so. I'd take 100 low-risk vulnerabilities that can be rolled into standard system updates any day over 12 highly critical vulnerabilities that need to be patched post-haste.

In order to judge the security of one operating system over another requires a lot more than mere vulnerability counts. You need to take a look at how these vulnerabilities can be exploited: can they be remotely exploited? What is the result of compromise? Is there anything in the operating system that mitigates the risk to other applications or system processes? And how actively is the operating system being targeted?

That last point is especially important when judging the overall "safety" of using any operating system. An obscure operating system isn't going to be targeted by active exploits and malware as often as the 90% plus industry giant.