Fighting Fileless Malware, Part 1: What Is It?

Even after 40 years of working to mitigate fileless attacks, the software industry is still struggling to eliminate them. By hijacking the control flow of a running application by exploiting a buffer-overflow vulnerability, fileless malware is responsible for numerous zero-day attacks. Yet despite the attention that Web attacks (such as injection and cross-site scripting attacks) get in the media, fileless malware remains the most dangerous cyberthreat today — and one few people understand.

Soon they will, courtesy of this three-part series in which I'll explore the software industry's attempts at solving this problem, including how countermeasures are being circumvented and what to do about it. In this, the first installment, you'll learn what fileless malware is and why it is so dangerous.  

What Is Fileless Malware?
Fileless malware hijacks legitimate programs via stealth attacks that evade detection by most security solutions. Because it doesn't rely on files and leaves no footprint, fileless malware is challenging to identify and frustrates the most adept forensic analysis.

A fileless attack uses a carefully crafted string of instructions — known as the payload — that is Base-64 encoded in order to evade checks that prevent malformed inputs. This payload can be delivered to the target host in many ways, such as in an input field exposed on a website, in a link, in a packet transmitted over a communication protocol (TCP/IP, HTTP, WebRTC, RTP, DNS, etc.), or in a script embedded in a file.

The payload then exploits a buffer-overflow vulnerability in a running process on the target system. This running process could be any server deployed at the edge connecting the organization's internal network to the Internet, such as a Web server, mail server, DNS server, SSH server, or any other kind of daemon. A daemon is a perfect target for a hacker because it is a long-running program that automatically restarts when it crashes and automatically reboots the application. Attackers can leak information about the target program in order to hone their attack with each crash and reboot, until the attack is successful.

Crafting a Payload
The hacker crafts this payload to hijack the victim application by subverting the return address of a function on the application's stack. By modifying the return address of the function with the vulnerability, the attacker can redirect the running application process to a different location on the stack when the function returns, thereby taking over the logic flow of the process.

After the process is hijacked, the attacker's objective is to quickly launch a terminal shell. Once the subverted process launches a terminal shell under the privilege level of the victim application, the attacker can use all the commands available in the system to do as he or she pleases.

The attack is effective because it runs covertly in memory under the running process of a legitimate application, without needing to create or modify any files on the file-system. If the system is rebooted, all traces of the attack disappears. Fileless malware evades nearly all traditional security solutions, making it very effective hacking technique.

Security practitioners generally think of security in-depth. There are multiple layers securing a network, whether it's on-premise or in the cloud. Since the large majority of cyberattacks are launched remotely, the attacker hides in the shadows of the Internet. For the attack to be successful, the payload must traverse the following perimeter security solutions:

Despite all these layers of protection, fileless cyberattacks remain rampant. Why? As long as traffic is allowed in and out of the network, it's a vector hackers can leverage to deliver their payload. Whether it's an email with a link or attachment, a Web service that has input fields that users can submit data, or an SSH daemon enabling users to remotely connect to a server,  the possibilities are endless. This makes it extremely difficult for security experts to defeat them. Once the payload crosses these perimeter security solutions — and it's not hard — it's game over. There's very little protection on the target other than, say, a signature-based antivirus solution to protect the operating system.

Next Week: Researchers have published and demonstrated how easy it is to circumvent the countermeasures widely adopted to block fileless malware. You'll gain a working understanding of how these countermeasures work next week, followed by an overview of the latest techniques to fight fileless malware in part 3.