Lack of Skills, Maturity Hamper Threat Hunting at Many Organizations

RSA CONFERENCE 2021 - Many organizations that have implemented a capability for cyber-threat hunting aren't deriving the full benefits from it, either because they lack the required skill set or because they have not fully made it a part of their cybersecurity program.

Tim Bandos, CISO and vice president of managed security services at Digital Guardian, says common mistakes that companies make include underestimating the time required for it and failing to get top-down support. Bandos hosted a session on best practices for threat hunting at the RSA Conference this week.

"Threat hunting is a critical component of an overall cybersecurity strategy whether it is done internally or through a managed service provider," Bandos says. Instead of waiting for an incident to happen, threat hunting offers a way for organizations to proactively seek out potential issues by setting traps and looking for behaviors within the environment that suggest suspicious activity. "But unless it is an official part of your program, you can't be successful at it," he says.

Interest in proactive threat hunting to stay ahead of new and emerging threats has grown in recent years. Security researchers have described it as giving organizations a way to try and uncover threats that may have slipped through or bypassed intrusion detection and prevention controls. The idea with threat hunting is to assume a breach has already happened and then track down all the different ways it could have happened using the same techniques an attacker likely would have used to pull it off. The focus is not so much on chasing down known threats alone but also uncovering new ones.

Gartner has previously described cyber threat hunting as useful, especially for organizations that already have maximized their alert triage, detection, and response processes and are looking for additional improvements in their security posture.

Bandos says threat hunting is something that organizations need to do on an ongoing basis using resources like MITRE's ATT&CK framework as a starting point. The framework offers different techniques and subtechniques that threat actors typically employ as part of the attack chain. Security teams can learn a lot by searching through their environment for signs of any of these techniques being used to enable or obfuscate malicious activity. "You can literally dive into a single one of those techniques all week searching through your environment," he says.

Similarly, organizations can learn a lot sifting through logs from their endpoint environment or by profiling all accounts that might have been created over the past week and separating the legitimate ones from the ones that might be more suspicious.

Successful threat hunting requires an awareness of new and emerging attacker tactics, techniques, and procedures. Equally, it requires a willingness to keep constantly going back and looking at older techniques, too, because attackers often tend to stick with tactics that they are familiar with and that have worked for them previously.

To conduct effective threat hunting, security teams need to have a reliable data source, such as a security information and event management system with centralized logs from multiple sources. Even logs from individual environments — such as endpoint detection and response, antivirus tools, networks, and data loss prevention (DLP) systems — are sufficient for threat hunting. Once the data source has been defined, threat hunters need to search through it using different techniques.

For example, the goal could be to hunt for signs of credential dumping in the environment. "You want to aggregate all threat intelligence that you know about credential dumping programs and commands and build a playbook around what you are going to actively seek around those logs," he says. The same approach can be applied for each of the different attack techniques listed in frameworks such as MITRE ATT&CK.

"I would start by focusing on a particular technique and then kind of exploding from there to a point where you can collect a particular artifact from every single endpoint in your environment and hunting through that data," he says. "That is when you start leveling up your capabilities in the threat hunting space." As one example, he points to the application compatibility cache that is stored on all machines. The caches contain a record of all processes that ran on a particular machine. Organizations can conduct a whole hunting campaign around just that data source alone.

The Right Skill Set
To be good at it, threat hunters need a solid understanding of security architectures, asset security, application security, and other fundamentals. They also need some level of incident response skills, including log analysis, malware analysis, forensics, and threat intel handling. Additionally, a threat hunter needs to be analytical, patient, and relentless, according to Bandos. The job can be tedious, and those without the right attitude can get quickly frustrated, he says.

One challenge with threat hunting is measuring success. It sometimes can be unclear if a threat hunting exercise turned up nothing because the exercise itself was not conducted properly or because there really was nothing to uncover. In smaller environments especially, threat hunters might often not uncover any new or hidden threats.

But in larger environments, with tens of thousands of endpoints, threat hunting can frequently uncover artifacts that might have slipped through intrusion detection and prevention controls. When threat hunting is handled internally, the analysts in charge most often tend to wear multiple hats. While that by itself is not a bad thing, it's important that threat hunting not be viewed as a part-time endeavor, Bandos says.

"The worst mistake is just assuming something is legitimate," just because it appears to be that way, Bandos says. Often, security analysts — especially the less experienced ones — can look at some activity or log data and ignore it because it appears to be normal. He adds: "Bad actors do a fantastic job of blending in and staying within the lines."