How Enterprises Can Use Big Data To Improve Security

[The following is excerpted from "How Enterprises Can Use Big Data To Improve Security," a new report posted this week on Dark Reading's Security Monitoring Tech Center.]

Enterprises have been leveraging big data tools and technologies to analyze everything from consumer buying patterns to competitors' product strategies. It was only a matter of time before the security industry took notice at the effectiveness of big data for analyzing complex and varied types of data inputs and sought a way to make it work for today's security problems. And now, it's nearly impossible to hear about a new security product without mention of big data.

Security vendors are promising that big data will pick up where security information and event management has failed to meet enterprise needs in keeping up with the overwhelming amount of data and new sources of information that need to be analyzed. The difficulty from a consumer perspective is that it's nearly impossible to cut through the vendor hype and understand what is meant by big data. And does a vendor's definition of big data fit with what big data really is?

In this Dark Reading report, we will cover what big data is so you can weed through the buzzword-ridden marketing, clearly see the areas in which traditional SIEM and log management systems are failing, and determine how big data tools and technologies can help turn many disparate sources of information into actionable intelligence.

No discussion of big data would be complete without mentioning the V's of big data: volume, velocity and variety.

Volume refers to many terabytes, and even petabytes, of information that must be processed. Velocity is the ability to ingest the large amounts of incoming data every second. Finally, variety relates to the different sources and types of traditional and nontraditional data being fed into big datasystems -- things like content from social networking sites and third-party threat intelligence services that don't fall into the old standard syslog and netflow formats.

The three V's are relatively standard across the different definitions of big data, but some definitions include a fourth V: veracity, or the trustworthiness of the data. For most data types, this is an irrelevant attribute, but with the variety of data being included in big data analysis, veracity is definitely something that should be considered, depending on the source.

For example, a NetFlow record from an internal router will have a higher veracity score than a blog comment, Facebook status update or post on Twitter. The difficulty lies in how much weight to give those sources compared with others, and then to provide context to the analysis. While some of that can be filtered and adjusted using complex big data processes, there will still be reliance on a skilled analyst to make a decision on the next action.

With SIEM, there was still the issue of the old adage "garbage in equals garbage out," but big data tools may be more capable of adjusting and adapting to the relevance of the data instead of blindly accepting it as fact.

Big data is being heralded as a technology that will bring new visibility into not only what's going on within a company's network, but also how external data sources can help predict upcoming attacks. Does this mean the death of SIEM? Some experts have sounded the death knell for SIEM, while others see the fusion of big data technologies and SIEM as the next evolution, taking security analytics to the next level. It's still too early to tell, but if the effects of big data on marketing and sales are any indicator, this is the kick in the pants SIEM needs to move forward.

The biggest changes big data can bring to SIEM will be better scalability and performance, the ability to include and analyze new types of data, and an increased speed of analysis so that decisions can be made more quickly. However, the combination of big data and SIEM cannot solve all of SIEM's issues and will still fail some of the customer perceptions of what's possible through the use of SIEM and big data tools.

To read more about how big data analytics may change the practices behind SIEM and security forensics, download the free report.

Have a comment on this story? Please click "Add a Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.