Finding Rootkits By Monitoring For 'Black Sheep'

A distributed system of monitoring groups of computers using the same operating-system configuration can detect the changes wrought by rootkits following infection, a group of security researchers from the University of California at Santa Barbara reported in a recent paper.

Inspired by the homogenous nature of corporate networks, the computer scientists developed a system, dubbed Blacksheep, that can monitor the kernel memory dumps of a large number of systems for changes that may indicate a compromise. The technique, which requires no signatures or foreknowledge of the attacker's code, could help companies detect attacks that other defensive measures fail to identify, says Christopher Kruegel, associate professor in the Department of Computer Science at UCSB and a co-author of the research paper on the system.

"We are not solving the general malware problem, but against the important crop of kernel-level rootkits and kernel-level modifications and exploits, it is a very powerful and very robust and general tool," he says.

The research (PDF), presented at last month's ACM Conference on Computer and Communications Security, demonstrated that in a cloud provider's network of virtual machines, the technique works extremely well, but it has significant challenges to overcome in a real-world network of employee workstations.

Rootkits are programs that allow an attacker to retain control of a compromised computer, even if the system is restarted or, in many cases, reimaged. The Stuxnet attack discovered in 2010, for example, used a kernel-level rootkit to stay persistent on infected machines. While attackers continue to improve rootkits with evasion techniques capable of dodging most defensive measures, a technique like Blacksheep detects changes to the kernel in machine memory and does not rely on signatures, says Giovanni Vigna, a co-author and professor in the Department of Computer Science at UCSB.

"The usefulness comes from the fact that it is not based on signatures and it's not based on the behavior of a piece of software," he says. "It's just based on the fact that, hey, all these machines should have a very similar configuration in the kernel, so if somebody is an outlier -- it might not be a compromise, maybe it is a malfunction of some sort -- but it's something that should be looked at."

[Insight into key characteristics, behaviors of cybercrime versus cyberespionage attackers can help -- but the threats aren't just from China and Eastern Europe. See Profiling The Cybercriminal And The Cyberspy.]

Blacksheep compares memory dumps from each monitored system, first creating lists of kernel memory modules that are then sorted and compared, calculating the distance that each list of modules is from the others. The system then compares each byte of a modules' code with other systems to find differences that could indicate changes inserted by a rootkit. Blacksheep also conduct memory crawling to catch changes to kernel data and checks five different kernel entry points for signs of changes.

The system detected all incidents of kernel rootkit infection on 40 virtual machines running Windows 7 with no false-positives. In a second test using physical systems installed with Windows XP, Blacksheep detected 75 percent of the rootkits with a 5.5 percent false-positive rate. The false-positives were due to the fact that real-world collection of memory dumps takes time, during which the kernel memory can change or become inconsistent, UCSB's Vigna says.

"Those inconsistencies show up as inconsistencies in our model, as well," he said.

Some security experts questioned the real-world applicability of the system. The technique seems interesting, but will likely produce a large number of false-positives in real-world scenarios since companies rarely have truly homogenous systems, says John Prisco, CEO of Triumfant, which has developed a system that detects changes to computers in a network, as well.

"The problem with these systems in the past is that you get 10,000 changes, and the model becomes confused," he says. "We tend to look at this as a data-mining analysis and try to net out all the things that are not relevant."

Yet the Blacksheep system does account for many of the failings of previous systems, such as integrity checking and invariant-based detection.

In addition, Blacksheep does show that research into gathering information among communities of computers is a valuable way to better protect the whole against threats, says Jerome Segura, senior security researcher, Malwarebytes, a software security firm. Antivirus firms and other security companies use threat communities to detect attacks against a member that may be repeated against other community members.

"The thing about a community is that you are getting information from resources you might not normally have access to, unless you had a big budget," Segura says.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.