Data to Drown In

I used to work for a dotcom as an IT security guy. It was a great job, and during the cash-burning days I got all sorts of awesome toys to play with. At our peak we had a team of three staff, two of us technical, plus at least one intern. This gave us tons of human power for a relatively small amount of data. I built us a nice Snort-based intrusion detection system (IDS) and enjoyed constantly tweaking it, putting new sensors out there, and writing automation tools for things like purging old entries from the database. It was great fun.

Then the bust hit, and we went from a team of five people to two, then just me. That was certainly less fun, and all of a sudden I started really realizing how much data I needed to keep on top of. We got in touch with what was then a very small startup named ArcSight Inc. , and started working with them on their product, which would bring in all of our data from the myriad sources, put it into a central database, and let us do good things like alerting us when certain data-driven rules fired.

This system was an early version of at least the reporting part of today's Enterprise Security Management systems, which are all about letting a small team of security people manage the huge volumes of data that are flowing from the increasingly common (and chatty) security devices. At the company I worked for we also included Windows event logs, Linux syslogs, and some router and switch events. This was a huge amount of data. While it was somewhat helpful to have access to all this data, it was also extremely difficult to make sense of how it all interacted, but the rules we used did help in that regard.

This is the real trick in security management, as it turns out. It is a relatively simple task to get all the data in a big database. Install agents on systems that can't send it out by some standard (or quasi-standard) reporting mechanism. Create listeners for all the (quasi-)standard reporting mechanisms. Write some parsers to break out the useful bits before creating database records. Synchronize clocks across the enterprise... You've got your data repository. None of that is rocket science.

The hard part is then dealing with all that data, writing the rules that will provide staff with the alerts they need. If you've got some good, reliable IDSs and firewalls, you can perhaps just use those systems as drivers. Find an interesting event from one of them, grab data from all sources on your network, and analyze. You'll probably want to include a rule or two on the routers so you can handle suspected denial-of-service issues, or at least flooding, even if it doesn't amount to a DOS attack. You'll still miss some attacks, particularly if they are tuned to avoid your IDS, but at least you'll get most of them.

I just read about of a parallel situation in Windows Vista, which will sound familiar to those of you who read my column on firewalls a few weeks ago. (See Sound the Alarm.) The new feature, named User Account Control, by all accounts, is plagued by an inability to present the user with a small set of meaningful, clearly written, descriptions of events that pose a real threat. This will, as I mentioned before, inevitably lead to the user either turning the feature off, or simply getting annoyed and paying no attention to the warnings. This is all part of the psychological aspect of security, as detailed in a recent Bruce Schneier interview. (See Schneier: In Touch With Security's Sensitive Side.) Most Vista users aren't going to know more about security than XP users.

It seems that Microsoft has confused the demand for security with the demand to turn all users into IT security pros. Rather, it's really a demand for users to not need to think about IT security. This is a persistent failing in security products, particularly for home users. We can't fix the users. They don't think they are broken. We need to find creative and, dare I say it, innovative ways to address these issues.

How are you all handling your volumes of security data, either at home or on the job? Do you have a fantastic ESM system? Something you built from MySQL, PHP, or seaweed and duck tape? I'm sure many of us are interested in finding out what actually works for others. Ping me here and let us know.

— Nathan Spande has implemented security in medical systems during the dotcom boom and bust, and suffered through federal government security implementations. Special to Dark Reading.