Hacker Smackdown

A showdown is brewing between two sets of security researchers over whether virtualization-based rootkits are detectable in a system or not.

A trio of researchers has publicly challenged Joanna Rutkowska, who made a name for herself at last year's Black Hat USA conference for cracking the kernel in Microsoft's Windows Vista beta release and since has led groundbreaking research in stealth malware, to let them prove that they can indeed detect her homegrown stealth virtual machine code called Blue Pill. They have offered her two shrink-wrapped laptops of her choice, one of which she would infect with Blue Pill -- if they can't find the laptop with the stealth malware, she gets to keep the machines. (See Rutkowska Launches Own Startup, Black Hat Woman, How to Cheat Hardware Memory Access, and Hacking the Vista Kernel.)

But the contest, proposed to take place at Black Hat USA in July, probably won't materialize -- at least in the near-term. It was the brainchild of Thomas Ptacek, co-founder and researcher with Matasano Security; Nate Lawson, researcher at Root Labs; and Peter Ferrie, senior researcher at Symantec, to disprove Rutkowksa's claims that there's no way to detect this type of malware.

In response to their challenge, Rutkowska has stipulated a six-month window to further develop the current version of her Blue Pill code, which she says is more a prototype than a "commercial grade rootkit." She also asked for five machines rather than two to eliminate the 50 percent chance the researchers could basically guess the infected machine.

Rutkowksa maintains it would entail two people working full-time for six months to get the code in the commercial-grade shape it would need to be in to win the contest (her original Blue Pill code is owned by COSEINC, her former employer). So she wants the standard industry fee for that type of work, which she estimates at around $200 per hour per person.

Ptacek, Lawson, and Ferrie have built their own detection tool that they say will definitely detect the Blue Pill, and with or without the Rutkowska challenge, they'll demonstrate the code at Black Hat.

"I'm looking forward to seeing their Black Hat speech -- or that they agree to our challenge -- and to see that they have actually some good, elegant method that really can't be defeated and that doesn't depend on specifics of a particular processor model," Rutkowska says. "Because if their detection method depends on processor implementation details" which Symantec's Ferrie has proposed, it's not practical because it's basically a commercial-grade detector, she argues.

Ptacek, Lawson, and Ferrie contend that virtualization-based malware is actually easier to detect than a normal (non-virtualized) rootkit because basically by definition it leaves a trail, introducing changes in the system's CPU clock, for instance. And the malware would have to be bug-free to truly emulate a system, anyway, Ptacek argues. "The problem with virtualized rootkits is... They have to present the illusion they are talking to real hardware and that's not an easy task," he says. "In order to do that, you have to write a bug-free program whose job it is to emulate bugs. And we don't know how to write bug-free programs."

He and his team say Rutkowksa's response to the contest is basically a no go: "She's basically saying 'no' " to the challenge, Ptacek says. "My read is that she's passively acknowledging that it's going to be totally detectable right now, so the only way to come through with the challenge is if we paid her tens of thousands of dollars."

Lawson says it basically proves his group's point. "We've been able to build the detector software in our spare time while keeping up a busy work schedule. We're so confident in it that we can issue this challenge and back it with hardware we bought."

The disagreement between the researchers has been as much fun as a heated debate, however, Ptacek admits. "We think this is what makes it fun... This is the way researchers do work together," he says. "I have a lot of respect for the work [Joanna] has done. She's hardcore."

Still, neither side is showing signs of backing down.

"It seems to me like their reasoning is this: 'We do not know how to do something, [so] assume others can't do it either,' " Rutkowska says.

Lawson promises that with or without the challenge at Black Hat, he and his colleagues will prove in their briefing there that "the detector always has the advantage."

Meanwhile, Rutkowska has another challenge for Ptacek and company: to post the source code for their detector tools. "If I were them, I would simply post the source code for their mysterious detector," she says.

But don't look for that to happen until Black Hat, when by the conference rules any code demonstrated in the briefings must be made available to the public.

— Kelly Jackson Higgins, Senior Editor, Dark Reading