The Biggest Threat? It May Be You

Find out about more about virtualization and the cloud in our digital supplement, part of InformationWeek's Green Initiative to reduce our carbon footprint.
Download Now
(Registration required)

We're halfway through 2009, and still no reports of production hosts being hyperjacked, leaving servers at the mercy of a compromised hypervisor. So what about all those dire 2007 predictions of virtualization-fueled havoc from third-party APIs, virtual NIC drivers, guest escalation breakouts, or compromised hypervisors? Anyone?

The only real-world vulnerability related to virtualization that's been reported for a major vendor was Microsoft's hypervisor privilege escalation vulnerability on the embedded hypervisor running Xbox 360s from the 2006 model year. No vulnerabilities have hit Hyper-V or Virtual PC.

As virtualization has gone mainstream and virtual machines have sprawled across data centers over the past few years, IT and security pundits repeatedly raised the bogeyman of compromised hypervisors. Black Hat seminars continue to debate potential risks and exploits. Gartner predicted there would be a hypervisor vulnerability worthy of a patch by the end of last year. Did any of the 19 security patches that VMware released in 2008 count as patch-worthy? Of course they did. Whether any of them patched a likely real-world exploit is up for debate.

The reality is that virtualization is simply software. Tightly written bundles of highly efficient code, designed with a hard crunchy shell, but software nonetheless. As with any complex, widely adopted program, there have been code and design flaws, and there will be more.

Given that, what are the likely threats? Virtual servers are just servers. The hardware abstraction and flexibility inherent in virtualization yields untethered VMs that are easy to create, deploy, shelve, and, well, even lose track of. The biggest threats to your virtualized world aren't bad guys wielding BIOS viruses, mythical blue pills, or a dastardly new method of usurping control of underlying host platforms. Rather, the weakest link is your own lack of planning, care, maintenance, and governance over your Wild West, devil-may-care VM farm.

Shore Up Standard Defenses
There's a long list of security concerns in small and large virtualized shops. You should be concerned about potential exploits allowing guest VMs to break out of their jails and into the host or hypervisor tier, says Greg Shipley, CTO at information security consulting firm Neohapsis, but you should be more concerned about unpatched guest VMs lurking forgotten on a test host or shuttling from host to host via poor live migration rule sets. Hyperjacking and intrahost risks are concerns, "but is that the right battle to fight?" he asks.

Just about everyone can benefit from basic risk management thinking, where the likelihood of a threat is plotted against the potential impact and remediation effort and cost, Shipley says. Most companies have basic security design and infrastructure concerns that arch over physical and virtualized environments. As boring as it sounds, addressing those security concerns will have a greater impact on overall safety than any single-purpose VM tool. Put another way, focusing on VM-specific solutions is premature if your physical shop is at risk.

Any unpatched server is a security risk, and a test or production VM, with management authority delegated to a business unit or development team, is a serious risk. Even if you have an automated patch management system or a formal in-house patch management strategy, you probably aren't 100% certain that all your VMs' OS and application instances are up to date. Attackers will probe your network for common exploits, and an unpatched Win2K3 server with a known hole is an easy mark. Bad guys don't care if the server is physical or virtual. They're just looking for exploitable targets.

Traditional automated patching strategies don't take offline servers into consideration. In the physical world, a dark server is a dead one. But in the virtual world, suspended or archived servers can be updated via virtualization-aware patch management tools. VM templates and base images also have to be maintained and patched. Lacking these tools, shops should vet each VM to check for currency and compliance, applying all patches before releasing a guest server to production.

Minimize Single Points Of Failure
Assuming you have all your servers under control, it's time to address other common risks. If you're tempted to consolidate servers, keep an eye out for single points of failure. There's a good chance that you won't save money by running multiple production VMs on high-performing entry-level platforms that vendors have been offering for free. They come with limited or no live migration capabilities, which means your VMs are trapped within one physical host server. If it has a bad day, all of your VMs are offline, and you have a multisystem outage.

In the past, you could rationalize the risk of hardware failure in terms of the limited risk involved in a traditional one-server-per-box model. In the virtual world, the risk goes up as the number of guests per host increases. So IT shops that are trying to save money through consolidation must plan for redundancy.

Basic designs cluster at least three hosts to provide live migration options as well as added performance headroom for production peaks and flexibility for hardware upgrades, maintenance, and host patching without production outages. All virtualization vendors have been ramping up entry-level feature sets, but the only way to get true high availability is to pay higher licensing fees, cutting into the cost savings from virtualization.

Citrix's XenServer offers free XenMotion, with high availability as a functional add-on to Citrix Essentials at $2,750 per server, and Microsoft promises that Hyper-V finally will have live migration later this year. VMware's vSphere Essentials Plus for small businesses starts at $2,995 for three servers and provides high-availability functionality; stepping up to vSphere Advanced for $2,245 per server gives you general-purpose vMotion functionality as well.

5 Must Dos

Securing The Hypervisor

1. Make hypervisor patching your new religion
There's no excuse for being behind on patch revisions. This is a zero-cost, relatively painless process to mitigate an unlikely but potentially critical risk. 2. Automate and audit guest VM OS and security patching
Hypervisor vendors offer built-in tools and third-party toolsets facilitate patch maintenance for guests. Pre-patch snapshots and rapid-restore functionality minimize risk. 3. Control physical and virtual networking
Segment control and maintenance networks from existing enterprise data connections ensure that live motion and control channels are separated from mainstream data channels. 4. Follow your hypervisor vendor's hardening recommendations
VMware and Citrix offer white papers on securing their offerings, and Microsoft offers a beta Win2K8 Hyper-V Security Guide. Read them well, recognizing that not all suggestions will apply to your organization. 5. Keep VMs on a tight leash
When admin authority for servers is delegated outside of IT, they can easily go rogue. It's imperative to maintain an audit presence and never fully release control.

Look Behind The Curtain
A "hybrid threat" occurs when a compromised guest server starts to reach out and probe for other servers to exploit. Traditional security tools in a physical stack would detect the aberrant behavior on the wire and raise an alert. But because of the way virtualized network communications are designed within a single physical host, it's possible that hackers could leverage a compromised VM to probe other VM guests on the same host without raising network-based security alerts. VM-to-VM communications may never go off-host; intrahost probes need not touch a wire that a monitoring system could detect.

Virtual security appliances from a number of vendors are designed to sniff out such shady behavior, looking at virtualized network traffic that never hits a physical wire, and providing virtual firewall and intrusion detection and prevention. VMware has many partners under its VMsafe banner, providing a security-specific API with the goal of bringing visibility similar to what we're used to with traditional physical IP security.

We won't debate the merits of a type-one hypervisor versus a type two. Admins should be aware that type-two hypervisors, such as Hyper-V and Parallels Server for Mac, have the inherent concerns of a full-blown OS running under the hypervisor; they provide a broader base of attack for the bad guys thanks to the underlying mainstream OS and require more care and feeding from IT.

Citrix, VMware, and others tout myriad benefits of a trim hypervisor. While smaller shops may be tempted to run Hyper-V as an additional service on a full-blown version of Windows Server 2008 for GUI simplicity and the other niceties, there's something compelling about the CLI and minimal services presenting a trimmed-down attack plane.

Seize Control
All major vendors and a handful of third-party vendors offer centralized management tools for virtual farms. Administrators can control and view every VM in an enterprise from a single console, and these tools are moving cross-platform, allowing control of heterogeneous environments. But these consoles present a risk for conflicting management rights and roles. In the past, large organizations may have relied on physical access restrictions to control who could touch a given server. Virtualization removes those walls, and Web-based consoles can deliver a screen view to anyone's desktop.

VM administration and server admin roles should be clearly defined as each server makes the physical-to-virtual transition. Servers handling sensitive medical, financial, or other privacy-restricted data need to be clearly identified. Access to all utilities, hypervisor consoles, and CLI management should be well thought out, and IP access rules for control and management networks must be clearly defined.

Avoid Cross-Talk
In addition to assessing access to host management network segments, the control channel for VM migration should also be segregated from your primary data networks for performance and security purposes. Live migration, using either vMotion or XenMotion, is facilitated by syncing memory across two hosts to shuttle a running VM from one box to another. That data transfer should occur on a private, secure network analogous to an iSCSI SAN. Setting up network segment settings for migration is straightforward on all virtualization host platforms.

When it comes to the multitude of new products in the virtualization security market, Neohapsis' Shipley says he hasn't seen any worth spending money on. Most companies "should devote resources to nailing down standard operating procedures for their virtual environments first," he says. "Too many lack basic management, vulnerability, and patch management tools."

For companies that have all the basics in place, though, investing in VM firewalls or intrahost virtual security appliances could make sense.