Threat Intelligence's Big Data Problem

First in a series on the evolution of threat intelligence

Something’s gotta give: nearly three-fourths of enterprises today say they ignore security events because they’re overwhelmed by the deluge of alerts. And that doesn’t even take into account the firehose of threat intelligence data they’re funneling today, a new report shows.

Mega-retailer Target was the poster child for security alert awareness gone bad—the needle in the haystack Target dismissed was actually the clue that it was under a major attack in the fall of 2013. Nearly three years after that epic data breach, security events, alerts, and threat intelligence feeds are exploding in many enterprises hungry for hints that they are in the bullseye. The tradeoff is that this deluge of data is drowning security teams who must sift, separate, and correlate the real threats from the false positives or irrelevant information.

Security event overload alone is causing some dramatic fallout: more than half of all security events get ignored by IT security pros due to the overload of information, according to a new Enterprise Strategy Group (ESG) report that surveyed 125 IT security pros on the state of incident response in their organizations. Around 30% of those organizations say they also have some 11 different threat intelligence feeds flowing in as well, the Phantom-commissioned report—published today--found.

Threat intelligence data is all about helping enterprises block or protect against the newest threats by providing in-the-wild attack and threat artifacts and intel that companies can compare and correlate with their security. But for many organizations, the deluge of this type of information isn’t much help if they can’t triage and apply it effectively. 

The threat intelligence market itself is booming, growing at a rapid clip at 84% annually, according to new data published today by IT-Harvest. The threat intel market—which was at $251 million in 2015—is expected to reach more than $460 million this year, says Richard Stiennon, chief research analyst for IT-Harvest.

Threat intelligence platform products such as those of ThreatConnect, ThreatStream (now Anomali), ThreatQuotient, and BrightPoint Security, made up $61 million of 2015’s total threat intel market revenues, according to IT-Harvest. The market is on track to hit $1.5 billion in 2018 at the current rate of growth, according to the report, which includes a look at more than 20 threat intelligence vendors, including FireEye’s iSIGHT Partners, Cyveillance+LookingGlass, Digital Shadows, and Flashpoint Intel.

“I expect a lot of churn and also a lot of startups,” Stiennon says of the threat intelligence space.

Signs of churn started to show in the past month, with Norse Corp.’s mass layoffs and executive shakeout. Security experts attributed Norse’s plight more to its own internal managerial problems and lack of a solid product as well some weak analysis reports, rather than as a bellwether of the threat intel space.

‘Threat’ Rebrand

Meanwhile, recent moves by other threat intel vendors show signs of a logical evolution of making threat intel more useful and manageable.

Late last month, ThreatStream dropped the “threat” moniker and rebranded itself as Anomali, now focusing on not just delivering threat intel, but also prioritizing and matching it for individual organizations. Threat intel has its own big data problem, according to executives at Anomali, which now is filtering down indicators of compromise (IOCs) and other threat intel for security event and information management (SIEM) systems, which it says weren’t built to process millions of IOCs.

“When we started [out], the volume of threat intelligence coming from feed vendors and open communities versus now was more manageable. There were hundreds of thousands of indicators of compromise, and now there are tens of millions,” says Hugh Njemanze, CEO of Anomali. “We expect this year to [reach] 100 million IoCs. There’s been an explosion.”

That kind of threat intel volume isn’t conducive for most in-house SIEM tools today. “Even the most robust SIEM is not able to ingest more than 1 million IOCs,” he says. Anomali’s new cloud-based products basically match event flows with IOCs, for example, and then feed contextual information about the incident to the SIEM.

“We’re taking on the burden of discovery and matching and letting the SIEM do what it’s good at: analyzing the millions of events they are collecting,” Njemanze explains. Security operations center teams need to know which IOCs are relevant, so that’s what Anomali is offering.

Anomali still offers ThreatStream Optic, its threat intel feed, in addition to its new Harmony Breach Analytics and Anomali Reports products. “We still see ourselves as a threat intelligence player, but we’re radically shifting how threat intel can be operationalized,” he says.

“I’m convinced TI platforms like ThreatStream’s [Anomali’s] have an opportunity. I haven’t seen anyone targeting dealing with the data. Building a distiller takes the good stuff out, and turns the SIEM into a log manager,” IT-Harvest’s Stiennan says.

ThreatConnect, meanwhile, has upgraded its ThreatConnect platform to better integrate a company’s security incidents with threat intelligence. “The goal of my platform is to bring the two together: every data set and correlate it with events and incidents that are unfolding so human beings don’t have to look at the noise. Instead, the most important things bubble up to the top, based on the underlying analytics,” says Adam Vincent, CEO of ThreatConnect.

ThreatConnect has partnered with Splunk, Palo Alto Networks, and others, to integrate threat intel with an organization’s incident detection and response processes. Version 4.0 of the ThreatConnect platform also lets companies customize reports for all levels of users, including C-level executives who want to see a map of which regions are targeting their company, for example, Vincent says.

Threat intelligence is about empowering decision-making, he says. “It’s not the end goal in itself.”

So rather than a retailer looking at 100 events in the order in which they occur, the threat intel platform would flag and prioritize events that appear to be connected or related to other attacks in the wild. “It would say this event is important because it looks coordinated, and it’s against equipment that has known vulnerabilities,” Vincent says. “And it looks at what type of techniques and tradecraft the [attacker] is using ... As the [company] investigates it, they are collecting additional information that is going to inform their decision-making.”

Most security vendors now offer some level of threat intelligence, and there are several open-source threat intel feeds as well. “The challenge right now is to tell high-quality threat intelligence from low-quality threat intelligence. It’s tough to distinguish, given the abundancy of options” out there, says Oliver Friedrichs, founder and CEO of startup Phantom.

“One of the biggest challenges is how to reconcile all the various feeds and how to actually make sense of them. The threat intelligence platform space is really striving to solve that,” says Friedrichs, whose firm offers an automation and “orchestration engine” for an organization’s security tools.

Related Content:

Find out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.