Attackers' Toolbox Makes Malware Detection More Difficult

Sometimes the simplest techniques can foil the complex systems created by security firms and large enterprises to detect malicious programs and files. Putting malware to sleep, waiting for a user to click, or looking for the hallmarks of a virtual machine can set off warning bells and cause a malicious program to cease running, making analysis difficult at best.

At the Black Hat security conference earlier this month, two researchers from digital defense firm FireEye showed off the menagerie of techniques developed by attackers to detect whether their programs were being run by analysts in a virtual machine or file-based sandboxes. Software security firms and defenders must do better to detect and allow the inspection of malicious programs that use such anti-analysis techniques, says Zheng Bu, senior director of security research for FireEye and a co-presenter of the session.

"File-based sandboxes alone are not effective in detecting malware," Bu says. "The tools are only as good as the person that uses it or the system that leverages the tools."

Since 2005, the number of variants of malware that security firms need to analyze and recognize with their software has skyrocketed. To cull the truly new malware from the known variants, antivirus firms monitor files for malicious behavior and then use automated analysis systems and cloud-distribution platforms to analyze the new threats and then pass the pattern files back down to the customers' anti-malware defenses.

If attackers efforts make automated analysis, which handles the vast majority of malware, more difficult, then they break the model, says Dean De Beer, chief technology officer for ThreatGRID, a malware-analysis service.

"If the attacker can force the workload back onto the human analyst, then they are succeeding because that means that the more malware that can be produced in that manner, the less threats a security firm can detect," he says.

The techniques identified by Bu and his colleague, Abhishek Singh, also of FireEye, include pausing execution for a specific period of time or waiting for human interaction, attempting to detect if the current system on which the malware is running is a virtual machine, and running on only certain types of systems with specific attributes to rule out virtual machine systems.

[Malware writers go low-tech in their latest attempt to escape detection, waiting for human input -- a mouse click -- before running their code. See Automated Malware Analysis Under Attack.]

The simplest techniques can be the best. Because analysis systems need to analyze malware quickly, for example, they typically look for malicious behavior in the first few minutes. That means that techniques that pause execution for a certain amount of time or that wait for a certain action -- such as a mouse click -- can be extremely effective.

Yet security firms have their own ways to deal with those techniques. Anything from hooking into a sleep function and, essentially, accelerating time, to assigning a higher level of suspicion to any program that appears to sleep following installation can help defeat the techniques, De Beer says.

Moreover, these techniques are not yet in wide use because they do require more advanced understanding of analysis techniques and programming, says Liam O'Murchu, manager of security response for Symantec's North American operations.

"There are only a few people that are really innovating, and there are only a few that are following what's out there," he says. "It's among those people where we really will see a large pickup of these types of things being used."

While there are examples of the technique being used to escape detection for some time, such as in the case of Trojan.Upclicker, it requires a lot of effort to continually defeat the latest defenses, he says.

Yet the same is true for defenders, FireEye's Bu says. Any type of sandbox requires an expert hand to be truly effective against modern malware, he says.

"It is a technology, it is a tool, and it's only as good as the system or people who use it," Bu says.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.