How Malware Writers Cheat AV Zero-Day Detection

As an experiment, Kyle Adams wrote what he describes as "ridiculously obvious" malware that most major antivirus products ultimately failed to detect. The only AV product that caught his malware was the freebie AVG, whose code-emulation feature sniffed it out.

So Adams, chief software architect for Junos Webapp Secure at Juniper Networks, kicked it up a notch and reverse engineered AVG's code emulation engine. Then he was able to bypass AVG's engine, as well, but he also noticed what attackers could do in that situatoin. On Tuesday, Aug. 5, at BSides Las Vegas, Adams will demonstrate how he cheated various AV products and how AV's code emulation feature for catching zero-day exploits has some weaknesses of its own.

"You start to realize things [the engines] can't handle, and if you explore them, you start to uncover compromises they didn't recognize," Adams says. However, "I didn't find anything that let me compromise the [code emulation engine] simply by [having the malware] scanned."

He plans to expose some of the engine weaknesses and name some of the major AV products his malware cheated in the first phase of his experiment. "If you know how the engine works, you can get around it."

His research shows that code emulation needs to be improved in AV products. "In some aspects, how code emulation works could be grounds for new exploits."

Adams says he's definitely not the first person to cheat an AV engine -- bad guys do it regularly -- but he wants to educate the security community on the problem. "I'd like to show how the malware authors approach the problem. I'm going to show building a virus from scratch… and testing it against the AV products like attackers do as they try to figure out how not to get detected."

His research shows that code emulation and sandboxing aren't really working anymore. "Now you can start to attack code emulators and sandboxes" themselves. At least 10% of attacks are attempting this today.

The malware Adams wrote and will demonstrate is a command-and-control bot. The JavaScript malcode he wrote runs on Windows. "It could be put together by anyone."

What can AV vendors do to beef up their code emulation? For one thing, "they should start penetration-testing their own AV software."

Adams will propose other solutions for improving code emulation. And he says he's definitely not picking on AVG, which he tried to contact before his presentation. "This is not a jab against AVG, as they get enormous credit for including such a powerful tool in a free antivirus client," he wrote in an abstract for his BSides talk.