Virtualization Security Heats Up

In March, Gartner ignited the blogosphere by stating the obvious: Virtualization creates new attack opportunities. There's still lots of smoke billowing around, but only time will tell how much fire is behind it, and who's fanning the flames. Vendors of new virtualized security "appliances" clearly have a stake. But many enterprises are realizing they rushed headlong into virtualization without considering the impact on their data protection policies, so IT pros do have legitimate concerns over the amount of real estate that could be consumed by a successful attack on a hypervisor.

If you're squirming right now, the big question you want answered is: Just how risk-exposed are we today? After all, in that same report Gartner predicted that a patch-worthy hypervisor vulnerability would be discovered in a mainstream product before the end of 2008. These potential vulnerabilities fall into two broad categories. First, if you can escape a client OS and move into a host OS, you have access to the data on all the other client operating systems on that machine. And there are whole new realms of rootkits being designed to take advantage of virtualization technology.

"People have been working on breaking out of the guest OS in VMware for some time now," says Greg Shipley, CTO of security consulting firm Neohapsis and an InformationWeek contributor. "And having a hypervisor rootkit installed would be a serious threat to any org. However, I don't see the development of the rootkit being the big challenge."

It's the process used to deploy such a rootkit that really intrigues Shipley.

"What's going to require more effort: Researching a vulnerability that allows us to break out of a guest OS and gain control of the hypervisor layer, or going after an administrator and hijacking the credentials required to install the rootkit, just like any other application? If the task was on my plate, I know which route I'd go."

As for breaking out of the client image, consulting company Intelguardians demonstrated just such an incursion into the host OS at last month's SANSFire show. Details of the vulnerability aren't public, so it's impossible to know what the attack was successful against, but you can bet these researchers aren't the only ones in this race.

The lesson is that organizations now need to assume that a sufficiently motivated attacker is capable of such an exploit, and plan accordingly. Defense in depth and proper virtual machine layout and design, including not mixing VMs with different security postures and requirements on the same host system, are crucial.

To find out how prepared our readers are, we fielded a survey--and got some eye-popping results. We can't help thinking that the 43% saying they believe virtualized machines are just as safe and secure as traditional environments are whistling past the graveyard. Of the 384 IT operations and security professionals responding, a mere 12% have put formal strategies in place to protect their VMs.

Now, many say they're relying on their current IT policies and toolsets to manage and protect virtual servers, and that makes sense ... to a point. Virtualized environments do face the same operational threats and risks as traditional servers, but there are added gotchas, from intrahost threats to vetting third-party hypervisor driver add-ons to new checklist items for corporate information security policies.

Let's face it: If a traditional 1U server is compromised, you'll feel some amount of personal shame, regroup, assess damages, fix the problem, and move on. Most shops have strategies in place to localize internal damage, with secondary and tertiary lines of defense to safeguard against a cascade of compromised systems. Problem is, few network monitoring and management tools are up to the task of securing guest VMs. When a traditional server gets slammed and begins displaying erratic or suspicious behavior, alarms will go off. But how effective are your tried-and-true netmon tools if all machine-to-machine communication is occurring between VMs inside your "data center in a box"? How much time will the bad guys have to probe, test, and exploit intrahost weaknesses before you see what's happening?

And is the current level of high security anxiety swirling around VM-specific environments justified? It's getting there, but that's the nice thing about smelling smoke--it warns that danger's afoot.

FIRM FOUNDATION

To weigh theoretical risks as well as where new applications of old attack points are feasible, you need to understand the underlying design of virtualized hosts.

Virtualization creates an abstraction layer separating guest operating systems from underlying hardware, enabling multiple VMs to be hosted on a single server. Virtual machines may rely on trim hypervisors using small, privileged code bases as the foundation for this abstraction; the strength of this approach is that performance of hosted apps can reach near-native levels. Products targeting the enterprise server market, including VMware ESX, Intel VPro, Virtual Iron, and XenEnterprise, favor a hypervisor design.

Alternatively, desktop VMs and Microsoft's virtual server offerings use a traditional "fat OS" model, where guest VMs ride atop full-fledged hosting operating systems.

While hypervisors provide optimized performance and a reduced attack surface, they also bring new vulnerabilities to the party and so need to have security baked in from the beginning rather than added as an afterthought. The million-dollar question here: Is it safer to rely on the open source community to vet and test Xen, or are VMware and other vendors of proprietary hypervisors the best path to secure hosts?

"From what I've seen, VMware's QA is pretty darn good," Shipley says. "They look like rock stars compared with many other companies. How many patches has Oracle come out with this year? I lost track as they approached triple digits."

Meanwhile, XenEnterprise's upcoming 4.0 hypervisor will weigh in at a trim 60,000 or so lines of code, XenSource CTO Simon Crosby says. Less code equals fewer potential bugs. Moreover, XenSource, which recently agreed to be acquired by Citrix Systems, uses IBM's secure hypervisor technology, and XenEnterprise has endured the pokes and prods of the open source community, earning a Common Criteria Level 5 rating.

Chip designers and VM software vendors also are working to stay on top of the security struggle. Steve Grobman, Intel's director of business-client architecture, says Intel VT-X server and desktop virtualization offerings are designed from the ground up to strengthen security. For example, Intel's current VT-enhanced server chipsets offer three new layers of code privilege for virtualization on top of the traditional three layers of CPU code privilege.

Of course, VMware owns the enterprise virtualization market, and the company is feeling pretty secure.

"Design, testing, and implementation of VMware ESX server contrasts with traditional, larger-platform operating systems," says Mendel Rosenblum, a VMware co-founder and chief scientist. "VMware has been focused on security concerns from our first line of code. I am 100% confident that we will not have a hypervisor compromise due to a design flaw."

We certainly hope his certitude proves warranted, and indeed, vendors have been successful thus far.

That noise was us knocking on wood.

DOOMSDAY TIME

The worst-case scenario in a hypervisor-based hosted environment? Hyperjacking, where an exploit leads to a compromised platform, allowing criminals full access to all hosted guests on a given machine. In subverting the hypervisor, malicious software could easily disguise its presence from traditional security tools that reside in either hosted-OS partitions or on any software layer above the hypervisor.

The exploit situation is analogous to the threat of cloaked rootkits compromising a standalone server OS. If you own the hypervisor, you own all data traversing the hypervisor and are in a position to sample, redirect, or spoof anything you please. Without some form of fail-safe, guest operating systems would have no way of knowing they're running on a compromised platform.

This is the stuff of nightmares when you're talking large-scale virtualization platforms that offer 10, 50, even hundreds of hosted servers running on a single piece of hardware. The potential risk for loss of control and revenue is enormous.

The answer is to maintain the integrity of the hypervisor while building in multiple fail-safes so hosted operating systems can ensure that they're communicating with an untainted hypervisor as a bridge to the underlying hardware and external connections. To run an unmodified OS outside Ring 0, the hypervisor must intercept "forbidden" Ring 0 instructions and emulate them elsewhere--without the guest OS recognizing what's going on. Silicon makers are looking to help here; for example, newer Intel and AMD chips targeting the virtualization market are able to insert a new privilege level beneath Ring 0. Both provide new machine code instructions that work only at Ring -1, intended to be managed by a hypervisor. In this way, a guest OS doesn't have to be modified, and the performance penalty from emulation is reduced.

It's the hypervisor's job to convince each guest OS that it and it alone has access to the host server's physical resources, while juggling access to ensure that programs and data don't leak between operating systems. Additional layers of code privilege for virtualized platforms on modern chipsets allow vendors to reduce the impact of a misbehaving guest OS in the event of a security breach or errant application.

To further minimize the risk of a compromised platform intercepting guest communications to the underlying hardware, some form of transaction confirmation needs to be implemented.

The Trusted Computing Group's most widely adopted standard is TPM, or Trusted Platform Module. The TPM is a critical element for a trusted hypervisor, providing hardware-based trustable root certificates, a trusted location for performing measurements, and several registries where trust measurements can be stored. TPM hardware encryption provides a guaranteed method for guest operating systems to vet communications with the hypervisor.

The goal of TPM is to provide tamper detection and prevention; Intel's implementation, for example, offers trusted VM monitor whitelists on a hosted platform. TPM is enabled before any software is loaded and can provide owner confidence over the boot sequence and ensure the authenticity of each system element as it loads. In a nutshell, the TPM hands control of the platform to the hypervisor only after the hypervisor has been loaded into a known, trusted state.

These concepts sound familiar? In higher-end versions of Vista, Microsoft relies on chipset-based TPM to provide BitLocker functionality for encrypting data stored on local drives. Future Intel and Advanced Micro Devices hardware platforms also are slated to use TPM to forge trusted paths to attached peripherals, relying on it to create and store unique keys for hardware-level encryption of data paths. This encryption, in combination with validation of virtualization components, should make intercepting the TPM/hypervisor handoff more difficult, increasing IT's confidence that OS communications to and from the hypervisor are untainted.

THE SIMPLE THINGS IN LIFE

Like a person who obsesses about being hit by lightning while driving without a seat belt, it's the mundane dangers associated with virtualization that are most likely to bite you. For example, both fat and hypervisor-based hosts are at risk of having a guest OS compromised via traditional threat vectors and exploits. An unpatched or poorly protected public-facing server is at risk, period, whether it's running on a standalone box or as one of many VMs on a large hosted platform.

However, common sense dictates that an organization's exposure increases in tandem with its reliance on virtualization and server consolidation--the more VMs per platform, the greater the danger of an undetected intrahost problem spreading. For all practical purposes, intrahost threats are invisible to traditional off-box safeguards. External firewalls and other security tools cannot inspect or control intrahost traffic, where packets never leave the host to traverse wired infrastructure. Concerns common in the real world that are tough to catch in a complex hosted environment include extraneous or suspicious intrahost cross talk posing as legitimate traffic, which is indicative of port scans, virus behavior, or other malware, and direct (targeted) or incidental denial-of-service attacks impacting other guest VMs because of consumption of CPU cycles, input/output resources, or virtualized network bandwidth.

"The 'more eggs in one basket' risk, from a pure operational perspective, has less to do with evolving threat vectors than simply being no-duh IT," Neohapsis' Shipley says, adding that IT groups saw the same dynamic with early storage area networks. "Most organizations can manage this risk by designing for extra capacity, running through virtual server migration drills, and keeping up with patching."

The last item is a lesson worth repeating.

"Even though I think VMware has done a good job in reducing the attack surface, ESX/VI3 still an operating system, derived from Linux, and as such it needs to be patched," he says. "Problem is, patching ESX servers is a riskier and more intrusive proposition because you're not just taking down one OS, you're taking down all the OSes it hosts, too."

The good news is that we've thus far seen relatively few critical VMware patches.

LIVING SAFELY IN A VIRTUAL WORLD

For now, we're all biding our time, waiting on the first successful compromise of a production hypervisor or VM monitor. To make sure your network doesn't become a poster child, architect your host implementation to keep the potential attack surface as small as possible. Find out where third-party device drivers reside--within the hypervisor to improve performance or at a higher layer, taking a slight hit while reducing the security risk.

Disable unnecessary emulated devices and lock down extraneous features and unused services on both the host platform and guests. Remember, a virtualized machine is still a machine. While that may seem obvious, IT needs to approach VMs with the same diligence and care offered to traditional servers, including adherence to security policies and guidelines. Thirty-six percent of our survey respondents admit to having no IT security or protection plan in place, with 23% fessing up that their policies are works in progress. Considering that upward of 70% of respondents have deployed at least one host platform, it's clear that unpatched or unprotected virtualized servers represent vulnerabilities just waiting to be exploited.

Ensure that security concerns, permissions, and environmental settings are properly configured to follow VMs to new hosts--while environmental flexibility is a key advantage of enterprise-class offerings like VMware ESX, without proper planning the ability to move VMs on the fly can be a curse.

"Along the lines of reducing attack surfaces and general exposure, I've seen organizations moving their VMware management segments off from the rest of the network and restricting who and what can gain access," Shipley says. "Clearly, firewalls in the data center is a newer trend, but certainly not one that's being driven solely by VMware. The more progressive IT teams I've seen are really starting to think about the concept of 'least privilege' models when it comes to network segmentation, and organizations can reduce their risk profiles by being diligent about restricting access to the VMware management infrastructure."

Shipley also stresses that IT should never put a virtualization host machine in a position where it has to enforce network zones--for example, having an ESX parent host guest VMs in and out of a DMZ.

CAN YOU BUY YOUR WAY SAFE?

Innovative vendors of dedicated virtualized security appliances, including Reflex's VSA and Blue Lane's VirtualShield, are focusing on virtualization as a solution to security problems, rather than just another attack vector. While we prefer to reserve the "appliance" moniker for things with three-prong plugs, we realize we're fighting a losing battle: VMware is pushing the concept of "drop-in" dedicated VMs that are purpose-built and preconfigured to address specific security or management needs, and many others are rushing to this nascent space; while no big player has given formal notice, we predict traditional security vendors such as Symantec will soon enter this market in force with tailored offerings.

But is that a good thing for IT?

"I can't help but wonder if some vendors are simply looking at all the virtualization going on and saying, 'Hey, how do I sell secu- rity to all these VMware shops?'" Shipley says. "Part of the burden on users/ consumers is to discuss what the true threat vectors are in their networks, and then look to tools."

Still, no one ever got fired for buying something that keeps an organization from being the next TJX. The clever folks at VMware understand this and recently bought Determina, a company that sells a memory firewall that can protect against stack and heap overflow exploits. While that's a pretty narrow protection goal, it's an important one. The problem is that for some applications, the Determina memory firewall could put a good-sized dent in overall performance.

VMware also absorbed Determina's LiveShield, which can apply patches on the fly--no need to reboot the server, just apply the patch in memory. Certainly this is right up VMware's alley as the technology isn't too far from its own binary emulation system, which rewrites parts of executable code as it loads.

While the idea of patching a running OS or application sounds interesting, it doesn't alleviate the need to test patches before they're applied. Usually, it's that testing that slows down the process, not the need to bounce the server.

This is where Blue Lane comes in with its patch-emulation products (there's a physical appliance version as well as a virtual appliance for VMware). The idea is to catch incoming attacks and make the fix just as an actual patch might do, before the offending packet ever gets near the server. While the company has its share of naysayers, VirtualShield performed as claimed when we tested it in our Florida Real-World Labs. And it got another seal of approval this week from none other than Microsoft itself, which tested the physical appliance and found it fully interoperable.

In the virtual world, the combination of these three products makes a pretty good safety net. The memory firewall will protect against overflow exploits, while Blue Lane's technology gives IT the time it needs to properly test patches and, once tested, LiveShield lets you apply them on the fly.

Finally, expect VMs to take a growing role on the desktop, with notebooks and PCs designed from the ground up to support admin-locked VM partitions constantly monitoring all running areas, safely removed from user-installed malware or human-error compromises.

If we leave you with one piece of advice, it's to work to raise awareness. The last question in our reader survey was open-ended, asking if readers had additional concerns or opinions on virtualization security. Sure, we got the expected rants and raves for or against specific vendors, but a recurring theme was, "I didn't have any concerns ... until I completed this survey."

If knowledge is power, consider yourself armed.

Continue to the sidebars:
Google's Gotcha,
and
Desktops In Play?