Metadata Consolidation

In storage, there's always discussion about consolidation; taking all your dispersed storage assets and consolidating them to a single storage system. But there's a different kind of storage fragmentation going almost unnoticed in metadata, or data about data. Many applications create metadata -- backup systems, data movement or archive applications, data management applications, and content search applications -- but only a few consolidate them.Isn't metadata small? Why is this a problem? Actually, this data can be quite large. As much as 10% of the storage footprint can be metadata. It's not just the size, but also the unneeded redundancy. Take this archiving example: if you are running e-mail archiving that can identify the same instance of a file and only store one copy of it to save on space (typically called single instancing) and a file archiving product that can do the same, but those two products don't share a metadata database, then not only do you have multiple copies of the same file, you are creating two metadata databases that contain very similar information.

The result is multiple metadata databases; all of them consuming a measurable portion of disk begin to add up. This affects backup and recovery of those metadata databases as well as any DR strategies that would involve replicating these files. In addition to the capacity and storage efficiency ramifications, there's also the cost of administration. Different metadata databases likely mean different interfaces, with separate policies to manage each task.

Don't be fooled by larger manufacturers with solutions in the above categories. Just because their applications share the same logo doesn't mean that they share the same metadata. Often these products are purchased, not internally developed, and although integration is often promised it typically only comes via a merging of the graphical interfaces, if it comes at all. Look for companies that have developed all the software internally, typically younger companies. In the e-mail/file archive space Mimosa Systems is a good example and in the data monitoring and management space companies such as Tek-Tools or Cofio.

George Crump is founder of Storage Switzerland, an analyst firm focused on the virtualization and storage marketplaces. It provides strategic consulting and analysis to storage users, suppliers, and integrators. An industry veteran of more than 25 years, Crump has held engineering and sales positions at various IT industry manufacturers and integrators. Prior to Storage Switzerland, he was CTO at one of the nation's largest integrators.