The Big Data Is The New Normal

I get a lot of questions on big data. What is it? How are people using it? How do you secure it? How do I leverage it? I've been on the phone with three different journalists in the past couple of weeks talking about what security analytics with big data really means. Be it journalists, security professionals, IT, or management, big data is relatively new to the mainstream practitioner, so the questions are not particularly surprising.

What is surprising is just about every new database installation or project I hear about sits atop a big data foundation. The projects focus on data, looking at new ways to mine data for interesting information. From retail-buying trends, to weather analysis, to security intelligence, these platforms are the direction the market is heading. And it's because you can Hadoop. Cassandra. Mongo. Whatever. And it's developer-driven -- not IT or DBA or security. Developers and information architects specify the data management engine during their design phase. They are in the driver's seat. They are the new "buying center" for database security products.

Since the bulk of the questions I get are now focused on big data, I am going to begin shifting coverage a bit to cover more big data topics and trends. And I'll spend some time addressing the questions I am getting about security and uses for big data. Yes, I will continue coverage of interesting relational security as I get questions or new trends develop, but because most of you are asking about big data, I'm going to rebalance coverage accordingly.

And to kick it off, today I want to address a specific, critical point: Big data is all about databases. But rather than a "relational" database, which has a small number of defining characteristics, these databases come in lots of different configurations, each assembled to address a specific use case. Calling this trend "big data" is even a disservice to the movement that is under way. The size of the data set is about the least interesting aspect of these platforms. It's time to stop thinking about big data as big data and start looking at these platforms as the next logical step in data management.

What we call "big data" is really a building-block approach to databases. Rather than the prepackaged relational systems we have grown accustomed to during the past two decades, we now assemble different pieces (data management, data storage, orchestration, etc.) together in order to fit specific requirements. These platforms, in dozens of different flavors, have more than proved their worth and no longer need to escape the shadow of relational platforms. It's time to simply think of big data as modular databases.

The key here is that these databases are fully customizable to meet different needs. Developers for the past decade have been starting with relational and then stripping it of unneeded parts and tweaking it to get it to work the way they want it. Part of MySQL's appeal in the development community was the ability to change some parts to suit the use case, but it was still kludgy. With big data it's pretty much game on for pure customization. Storage model, data model, task management, data access, and orchestration are all variable. Want a different query engine? No problem, you can run SQL and non-SQL queries on the same data. It's just how you bundle it. Hadoop and Cassandra come with "stock" groupings of features, but most developers I speak with "roll-their-own" infrastructure to suit their use case.

But just as importantly, they work! This is not a fad. These platforms are not going away. It is not always easy to describe what these modular databases look like, as they are as variable as the applications that use them, but they have a set of common characteristics. And one of those characteristics, as of this writing, is the lack of security. I'll be going into a lot more detail in the coming weeks. Till then, call them modular databases or database 3.0 or whatever -- just understand that "NoSQL" and "Big Data" fail to capture what's going on.

Adrian Lane is an analyst/CTO with Securosis LLC, an independent security analyst firm. Special to Dark Reading.