Big Data Security Discussion

Last week I participated in 'a "tweet jam" to discuss security of big data clusters. That use of social media is pretty interesting as it's an open forum to any/all who have interest. It's a great way to get some community involvement and buzz, but at the same time, 140 characters is insufficient to answer complex questions. You can portray an idea, or a facsimile of an idea, but it's insufficient to flesh out the nuances of a subject as complex as securing bi data clusters.

Worse, I thought some of the other participants in the conversation provided tangential responses to the questions posed, possibly generating more confusion than clarity. So I wanted to provide a little more color and context to the questions posed about big data security during that chat session.

Q1: What is big data security, and is it different that traditional data security?
Big data security is about securing big data clusters and the data stored within the cluster. The types of technologies we use to respond to specific threats are not new; for example, we still use encryption to secure data stored on data nodes, and we still need to use authentication to identify users before they are granted access to the system.

However, big data clusters are a type of distributed application. The goal is to protect both the data stored in the cluster, but also the function of the cluster itself. This poses new challenges because of the way big data scales, the "velocity" of data that is input into the cluster, and the elastic, self-organizing nature of big data clusters. These core traits break many existing security tools. That is to say most security tools don't automatically work with big data. For most security tools to work as currently implemented, they limit some fundamental trait of big data.

Q2: What about security systems as producers of big data?
The question is backward. It's not about big data as producers of security data -- rather consumers. We have event data coming from every device, agent, and application deployed, and security information is embedded -- sometimes subtly so -- in the stream of normal activity. Why big data is viewed as a solution because it can handle the volume of data, the rate at which it is being generated, and provides the analysis engine to comb through the data and find actionable security information.

Q3: Most big data stacks have no security built in. What does this mean for securing big data?
The premise is only partially true. Every open source and commercial distribution I've seen offers some security. Usually logging, identity management, and application proxies used to secure client-cluster sessions. But that's only a partial tool set to combat threats to the cluster or to data privacy, and the implementations of these tools is usually poor. That means looking at what third-party options are available and don't break the cluster. Today, most use a security model where the NoSQL database is secured behind a firewall, and partial security controls are applied to inbound traffic. Most do not leverage the cluster itself for security, rather the existing network tools they have in place. They benefit for familiarity of these tools and cost savings for using existing products, but the cost is partial security model that

Q4: How is the industry dealing with the social and ethical uses of consumer generated data?
I think the jury is out on this, and it's too early to tell. I cited the case where Target determined a teen pregnancy before the parents did. That's one of the few times tangible social and ethical issues have been discussed. Right now, the companies I've spoken with feel "my business productivity trumps you're privacy," so businesses are "full steam ahead," and the social and ethical issues will get sorted out later.

Adrian Lane is an analyst/CTO with Securosis LLC, an independent security consulting practice. Special to Dark Reading.