Research Casts Doubt on Value of Threat Intel Feeds

Collect threat data from two of the largest threat intelligence providers, and the risk landscape they portray will be completely different — raising questions about the utility of threat intelligence feeds to organizations, a group of researchers said this week.

The researchers, from universities in the Netherlands and Germany, compared threat indicators from four open source threat intelligence feeds and two commercial feeds — which the researchers could not name — and found very little overlapping data between the services. On the commercial side, the larger Vendor 2 had 13% of the data covered by Vendor 1, while Vendor 1 only replicated 1.3% of the indicators from Vendor 2, said Xander Bouwman, a PhD candidate at Delft University of Technology and a primary author of the paper, in a presentation Wednesday.

"If two threat intelligence vendors are describing the same threats, you might expect that they are coming up with the same data," he said. "We find that this is not the case."

Even in tracking the same advanced persistent threat (APT) groups, threat intelligence vendors did not seem to collect the same data. Focusing on 22 threat groups that both vendors claimed to be tracking, the researchers found, at most, a 4% overlap in threat indicators, Bouwman said.

"This raises some questions about the coverage that these vendors are providing," he said. "If there is not so much overlap, what does that say about the visibility that these vendors are providing for the threat landscape as a whole?"

Threat intelligence includes open source threat intelligence, shared intelligence between organizations in the same industry, and commercial threat intelligence services. Open source threat intelligence often includes data from DNS blocklists, abuse feeds, malware hashes, and phishing lures. Shared intelligence is usually not available unless the organization joins a particular industry group. 

Commercial threat intelligence is often sold as a combination of reports to inform security teams and analysts and machine-readable indicators of compromise (IOCs) that be used to detect threats. A typical commercial feed, for example, could have dozens of threat reports and hundreds of IOCs every month. 

Unfortunately for potential customers, the uneven coverage means every threat intelligence provider's data set will be different, and there is little guarantee — or probability — that the threats will match what the customer will see. Without more information, the services are hard to evaluate, Bouwman said.

"This is what we refer to as a market with asymmetric information," he said. "The sellers know what they are selling, but the buyers don't know what they are buying."

The researchers compared the two commercial feeds with four open threat intelligence (OTI) feeds from Alienvault, Blocklist.de, CINScore, and EmergingThreats. While a few of the OTI feeds had significant overlap with other OTI sources, the commercial vendors had less than 1% overlap with any open threat intelligence feed. 

The lack of overlap raises questions about coverage and whether the services are providing a realistic picture of the threat landscape, Bouwman said.

Customers typically use threat intelligence for network detection, situational awareness, and prioritizing security operations centers' (SOCs) activities, the researchers found. Commercial feeds are better at providing context to users, according to a survey of 14 users of threat intelligence. Moreover, threat intelligence does not seem to be limited by cost, with only one in five in the survey citing cost as a factor. 

Unfortunately, customers are not very mature in terms of their knowledge of and skill in using threat intelligence, Bouwman said. Two respondents, for example, canceled their threat intelligence feeds because they were covering a sector unrelated to the organization's business.

"Customers do not seem to care about coverage, they are not optimizing for detection, and they are not talking about metrics," he said. "If they do mention metrics, it is almost always talking about false positives."

Overall, threat intelligence appears to be less about attaining insight into most threats and more about using the reports and IOCs as a way to understand the threat landscape, as well as occasionally for threat hunting. The most important factor may be whether the threat intelligence service helps save analyst time, the researchers stated.

Commercial vendors should help customers get the most productivity out of their feeds to justify their high cost, while customers need to require vendors to provide more information about the coverage the feeds provide, Bouwman said.

"In a market with asymmetric information, the willingness of consumers to pay might eventually go down because they cannot distinguish the good from the bad," he said.

Related Content: