Foxy Vista Henhouse

Common wisdom clearly states that allowing the fox to guard the henhouse is a bad idea, but in the case of Microsoft's entry into the security software market it appears that we may be allowing the fox to do something even worse -- design and build the henhouse. It is not very surprising that Symantec and McAfee have loudly and publicly decried Microsoft's security approach as anti-competitive and bad for security.

Security software firms are worried about Microsoft getting into their business, and probably within reason. Microsoft is a harsh competitor with huge market-moving power.

But many of the ideas that Microsoft is proposing for security are sound and could be very beneficial to security in the long run. The problem is that these good ideas have been tied (sometimes unfairly) to Microsoft's security software business plans.

In the final analysis, some of the anti-Microsoft noise is justifiable, and some of it is not. Untangling the two central threads can help us understand what's going on and form an opinion about what should happen next.

Protecting the Kernel is Good
The Vista kernel is designed so that interposing inside the kernel by hooking APIs or by otherwise patching critical system calls or swapping out critical kernel modules is disallowed. Technically, this is accomplished by signing kernel modules and enforcing the "no changes allowed" rule. This kernel-level protection comes courtesy of the Software Protection Platform which includes the Code Integrity (CI) verification subsystem.

The CI system is a cryptographic verification system that ensures that system binaries have exactly the same bits they had when they were certified at build time. CI also makes sure that no unsigned drivers are ever run in kernel mode (at least on 64-bit systems; things are a bit less draconian on 32-bit systems and mostly focused on supporting DRM for HD content). CI starts at boot time, but as we all know, boot loader security is notoriously tricky.

Make no mistake about it, this is an excellent security move by Microsoft designers. In order to support proper encapsulation (and leverage the basic idea of a trusted computing base), very strong boundaries must be identified and enforced -- especially in and around the kernel. This is usually best accomplished through the development of APIs that serve as front doors into the kernel, followed by very strong enforcement of API-only access controls. Compartmentalization of this sort is an essential security engineering best practice.

For many years, kernel-level APIs have been ignored and/or thwarted by malicious attackers and security software providers alike. Greg Hoglund and I describe highly technical ways of patching the win32 kernel found in Windows-NT and Windows-XP in order to thwart security in our book "Exploiting Software" (Addison-Wesley 2004). The hackers responsible for rootkits use kernel patching and DLL substitution methods as a matter of course. Given that rootkits are the apex of modern malicious software, coming up with a way to thwart this kind of activity makes perfect sense.

The problem is that security software vendors including Symantec and McAfee have used the very same techniques for years in the name of good. Antivirus software and personal firewall software pulls all sorts of fancy kernel-interpositioning kung fu.

The problem is that if Microsoft puts hard core controls around the kernel, the approaches used by security companies (and malicious hackers) will stop working. A complete redesign for the existing security providers is an expensive proposition that the security companies would rather not incur. Of course the malicious hackers would rather not see all their work go down the drain either!

Microsoft should go ahead and properly encapsulate kernel technology. Better security requires a more reasonable approach to OS design and API enforcement. Bravo to Microsoft for doing the right thing.

Being Paid to Fix Your Own Mistakes is Bad
All is not well, however. Microsoft's entry into the security market is hugely problematic and will be bad for security in the long run. The problem is simple and obvious. The reason we need security software like antivirus tools and personal firewalls is that OSes have traditionally suffered from all kinds of security problems (both bugs and flaws). Virus writers and malicious hackers take advantage of these weaknesses to create and spread malware. Viruses, worms, rootkits, and botnets are the result. This is a mess.

If you think about it this way, Symantec and McAfee exist to clean up the security mess that vendors like Microsoft made with their early, insecure OSes. Lots of money has been made by these companies, and they provide a very valuable service.

Here's the problem. Microsoft is still in the business of building OSes. They should concentrate their resources and security efforts on building better OSes. And to their great credit, they have been attempting to do so through activities such as the Trustworthy Computing Initiative and the software security work of Steve Lipner and Michael Howard.

If, instead, Microsoft is allowed to enter the security software business (note the critical difference between software security and security software!), they will in some sense be responsible for building cleanup technology designed to clean up their own mess. Then the bigger the mess they make, the more demand there will be for their janitorial supplies!

Microsoft may be too responsible to manipulate its security defect density intentionally in order to create demand for its security software, but the fact that this is even possible is a great worry. This is like allowing the fox to design and build the henhouse, not just guard it.

Microsoft is right to insist on controlling access to the kernel. Kudos to them for this move and for their progress in software security. But Microsoft should not be allowed to get into the security software business. They should leave that to other vendors. If both of these things happen, we'll all be better off in user land.

— Gary McGraw is CTO of Cigital Inc. Special to Dark Reading