Tech Insight: Microsoft's IPSec

Microsoft’s support of the IP Security (IPSec) standard was enhanced with the release of Windows Vista this year, and interest in the technology will likely grow with the introduction of Windows 2008. For smaller organizations, IPSec could prove to be a cheap alternative to other network access control (NAC) technologies, or a stepping stone to a full implementation of Microsoft's Network Access Protection (NAP) in large enterprises. Either way, it’s time for organizations to take a closer look at IPSec’s capabilities.

Since Windows 2000, IPSec has been included in every Microsoft Windows desktop and server operating system. As a staple of the operating system, it’s surprising that more companies don't take advantage of the technology, but many IT professionals still labor under the notion that IPSec is a VPN technology only used for remote connectivity.

"The knee-jerk reaction is that IPSec is used for VPN," said Microsoft’s Ian Hameroff in a blog. "We want to unlock the other value [in IPSec]." While IPSec certainly can be used in VPNs, it can also be used for basic packet filtering, or blocking solely based on source or destination IP, source or destination port and network protocol.

The real power of IPSec, however, is in its ability to protect managed Windows machines from non-managed machines by requiring authentication before network communications can occur between two hosts. This authentication is based on Kerberos, certificates, or pre-shared keys, and optionally, encryption can be enforced to secure communications between endpoints.

Microsoft calls this method of protecting managed endpoints and servers from un-managed machines "domain isolation" or "server isolation." The company has produced a significant amount of documentation on what it is and how to implement it. In 2004, Microsoft deployed domain isolation using IPSec within its own enterprise network, protecting over 200,000 systems.

There is a clear need for this sort of endpoint protection. In a survey published earlier this month, the Ponemon Institute and Deloitte & Touche found that 85 percent of enterprises have suffered at least one reportable security breach in the last 12 months, and a staggering 63 percent said they suffered between six and 20. (See Study: Breaches of Personal Data Now Prevalent in Enterprises.)

IPSec could prevent some of these breaches by simply stopping rogue machines from communicating to the managed Windows machines. Even malicious attacks that attempt to wrest remote administrative access from vulnerable Windows services would be prevented, because the connection wouldn’t be allowed without the attacking machine being part of the domain and authenticating first.

If your company is one of the many that are planning to implement NAC, IPSec should be an important consideration in your technology selection. Unlike other NAC solutions, Microsoft's NAP can quarantine hosts using IPSec in addition to DHCP, VPN, and 802.1x enforcement.

With NAP and IPSec, if a Windows endpoint does not meet the required health checks (antivirus installed and updated, latest Microsoft patches applied, etc.), it would only be allowed to talk with the NAP servers to begin remediation. Once the endpoint has passed the health checks, a health certificate server provides a certificate proving that the host is in good health. IPSec policies would then allow the "healthy" endpoint to communicate to other managed hosts.

So if IPSec is so great, why isn't it more widely used? One answer is its history. Besides being commonly perceived as a VPN-only technology, Microsoft's IPSec has been difficult to configure in the past. In fact, it previously, had to be configured independently of the Windows Firewall, which sometimes led to contradicting policies.

Recognizing these issues, Microsoft released the Simple Policy Update for IPSec in 2006 for Windows XP and Server 2003, and the company has combined the configuration of IPSec and Windows Firewall in Vista and Server 2008. Is it too late to change users' minds about IPSec? Only time will tell.

Windows-based IPSec also may be perceived as a Microsoft-centric solution that doesn’t extend well to other platforms, such as Linux and Mac OS X. In the case of NAP, that won’t be true for long -- Microsoft has more than 100 NAP partners, and several of them are working on NAP clients for Linux and Mac. If you want some examples, take a closer look at UNETsystem Co. Ltd. and Avenda Systems Inc.

Last May, Microsoft’s Open Source Software (OSS) Lab completed IPsec interoperability testing between Linux and Vista, which seems promising. In the test, the lab successfully established authenticated and encrypted communications between Linux and Vista endpoints using certificates and pre-shared keys. This testing could eventually make it possible for Linux systems to coexist in an IPSec domain or server isolation environment with a Windows host. So far, however, we haven't seen any similar testing with MacOS.

If your IT shop has looked at Microsoft's implementation of IPSec in the past and dismissed it, it’s time to take another look. The technology has been improved. The price is right -- it's already included in Windows at no extra charge -- and the added security of domain and server isolation is protection that could prevent unnecessary data breaches by rogue machines. And it's a great start toward NAC, which is already supported by Vista and will be included in Server 2008 and Service Pack 3 for Windows XP.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.