Can Machine Learning Overcome the Threat Intelligence Gap?

Feeds, risk lists, analyst notes, Dark Web sources, threat research factoids and more. Compiled into a kind of security encyclopedia for reference. But the number of volumes is too big, and the pages regularly go missing or fall out.

No one can really find what they're looking for. This is the threat intelligence (TI) challenge we all face. A fragmented deluge of information with only a few worthy nuggets within. And it's probably going to get worse before it gets better.

"The fundamental problem is scale," Matt Kodama, vice president of product at Recorded Future, a Somerville, Mass., TI firm, told Security Now. "Threat actors, malware tools, web-connected infrastructure and our threat surfaces of exploitable technologies are all growing much faster than security or risk teams, with no end in sight."

(Source: Pixabay)

Couple this with a shortage of TI talent in the market, which threatens staff turnover and loss of knowledge, and enterprise worries begin to multiply.

Fortunately, the emergence of TI in the last couple of years has begun to assist security teams at an operational level in understanding today's threats by providing a heads-up before a potential attack. But Kodama points to a lack of intelligence about potential threats at the strategic level that is stymieing long-term planning.

"TI should help business leaders to see and take tough strategic decisions that permanently reduce risk. The private sector needs threat teams that can support them at both operational and strategic levels," he said.

Intelligence challenges
The real issue is not that there's a lack of clear information; it's weeding out threats that have proximity to the enterprise, while also getting arms around data from a pool that is more like an ocean. There's a growing acceptance in the industry that machine learning provides support where humans are failing.

"Machine learning is used in collection of documents from web sources, for semantic analysis of text to identify relevant topics, entities, and events, and in predictive risk scoring models," said Kodama.

Currently, effective machine learning is enabled through training data, with security teams effectively baby-sitting until the technology is mature enough to fly the nest.

Not only are enterprises challenged by the scale of threat information, they're also struggling to manage multiple TI providers and platforms. Kodama believes that rationalizing reporting complexity and centralizing information from analyst notes with platform data are the answer.

"Every use case for threat intelligence has unique requirements," he said. "The wrong data integrated in monitoring and alerting solutions can inundate teams with false positives and noise. The ability to tailor the data stream for their specific needs and use cases enables organizations to have the threat intelligence they need where they need it."

Market response
The market for TI solutions includes the "have," and the "have-nots."

Matt Kodama, vice president of product at Recorded Future
\r\n(Source: Recorded Future)\r\n

For enterprises that have TI, there's a desire for more coverage and less complexity. These firms are going through a workload balancing exercise where freeing up TI teams to do more analysis work is the goal.

"They want faster, tighter collaboration between TI, SOC, IR and VRM without bringing in yet another security data technology to deploy and manage," said Kodama. Those without TI -- the majority of companies -- have a different set of requirements as they search the market.

These companies struggle with alerting on incidents outside their network, such as brand monitoring, exposed credentials, exposed source code or intellectual property. They want better quality indicators of compromise for detection controls. And, no surprise, they need more rapid, more relevant information in order to assess risk and potentially escalate to avoid issues.

Asked if adding an end-to-end solution means a substitution or "rip and replace" scenario where there's overlap, Recorded Future counters that actually, most organizations don't even have a threat intelligence platform yet. For those that have a more mature TI approach or team, the firm will work alongside existing SIEMs, incident response platforms, and security orchestration and automation platforms.

Related posts:

— Simon Marshall, Technology Journalist, special to Security Now