Popping the Vista Kernel

There are so many great things to look forward to in the new year, including the API spec from Microsoft for endpoint security vendors that want to develop compatible products for the 64-bit Vista kernel.

The specification, with implementation targeted for delivery with SP1, is all but guaranteed to stoke the debate between Microsoft and the security industry for the optimal way to secure Windows. The discussion revolves around Microsoft's implementation of Kernel Patch Protection (KPP), commonly known as PatchGuard, and the security vendors' reluctance to be constrained in how they secure Vista customers.

PatchGuard refers to the Microsoft technology in 64-bit Vista that prevents third-party software from hooking and modifying the Vista kernel. Pre-Vista security software, such as anti-virus, anti-spyware and host intrusion prevention, patched the Windows operating system, file structure and network stack. These hooks enabled third-party products to insert security logic into the flow of kernel-mode processing to detect and block attacks.

Microsoft could not easily change sections of the kernel without running the risk of breaking unsupported hooks for end-user security products. A fault in security code running in the kernel would lead to the ever-popular Windows blue screen, safe mode re-boots, and maintenance headaches for users. Microsoft's PatchGuard plan was to preserve the integrity of Windows by repositioning all third-party software outside of the kernel. Since Microsoft is the custodian of Vista, it is perfectly within their rights to do this.

Meanwhile, security vendors, such as McAfee and Symantec, insist that they need to operate deep in the kernel to protect consumer and business users. They feel strongly that symptoms of poor health are often only found in the kernel, and that low-level techniques are necessary to prevent the security system from being subverted by a sophisticated attack.

They further argue the need for extensive visibility of Vista internals to be able to perfect next-generation behavioral approaches for zero-day protection. The traditional vendors will also talk smack about unfair advantages, being better at security, and preserving customer preferences. While these may be justified in other areas, it is mostly hot air when it comes to PatchGuard.

The industry does need an interface that allows security vendors to innovate and make the Windows computing experience safer. Microsoft can't do it all, with behavioral methods and real-time interaction with the security infrastructure being two areas where Microsoft is still feeling its way. I'm willing to bet that over the five-year lifespan of Vista there will be more as well.

Microsoft should be developing an interface that allows registered third-party vendors to be able to inspect the most sensitive areas of the Vista kernel. Security products, because they snoop and inspect other processes, can be confused with attack software. Registration via a signed certificate by a functioning security vendor is an important and reasonable precaution. Everybody wants a supported interface; nobody wins with unsupported hooking practices.

The Vista kernel patching interface specification is causing angst in the boardrooms of endpoint security vendors. While there have been constructive discussions, nobody outside of Redmond really knows what features the interface will support. Let's hope that when the specification comes out, there will be a brief period for review and public comments, followed shortly by a frozen version.

Professional security vendors need to be able to innovate on behalf of Windows users. This is clearly a good thing for customers. It's about time that Microsoft and the security vendors stop their bickering and get their act together to secure both Windows and enterprise networks.

— Eric Ogren is a security analyst with Enterprise Strategy Group (ESG) Special to Dark Reading