Care and Feeding of Your SIEM

System information and event management (SIEM) systems and intrusion detection and prevention systems (IDS/IPS) are our frontline defenses in an increasingly hostile world. As effective as they seem, if they're not properly configured and maintained, they can become essentially useless.

Part of the problem is that this work isn't sexy or easy — it's basically digital sanitation. But it's extremely important. Imagine what would happen if your company's garbage suddenly stopped being collected. While SIEM/IDS/IPS maintenance isn't quite as stinky, it's very similar. Networks change. Threat landscapes change. Most IDS rule sets don't change (other than getting bigger). Software or services stop being used. And unfortunately, somebody needs to paw through your SIEMS's entrails to find these things.

There are a few ways to mitigate the grunt work to make the task more manageable for your security engineers.

1. Roll Out Slowly
If you are just rolling out your SIEM/IDS/IPS system, avoid turning on everything at once — this is a major mistake. Turn on subnets or VLANs one by one — tuning as you go. A SIEM throwing out 100 alerts in a day is a sure bet of burning out your security analysts quickly, and pretty much ensures overly liberal whitelists will quickly become commonplace.

2. Good Documentation
Make sure every rule set added is documented (and reviewed by someone) when it is added. And make sure the documentation is complete. The "magic whitelist" is routinely encountered when doing audits. Such an incomplete entry usually happens at 4:59 p.m. on a Friday afternoon filled with false positives. Unfortunately, Monday rolls around and the whitelist isn't updated, and months later a similar alert is triggered. Make sure your documentation is precise and helpful. Don't allow any ambiguity. If something is a false positive, explain why.

3. Fine Tune (Preferably the First Time)
Don't rush through the configuring of your baseline — no matter how long it takes. Tuning out the legitimate false positives while taking the time to thoroughly investigate each alert is important. The more granular you can get your rule sets, the better. If you can pinpoint false positives from specific machines in a subnet, exclude the machines and not the subnet.

Remember that you are doing this not only to find stale or overly liberal policies but to improve performance. Your SIEM/IDS needs to process every single rule it encounters, so the more rules you have, the slower your device will be. Look for ways you can consolidate rules wherever possible — and toss the chaff.

4. Frequent Auditing
Make auditing a regular exercise. Instead of leaving the entire task to be done once a year, make the process a monthly or bimonthly process. Tie changes into a change control process wherever possible. If changes are made on the fly, make sure they are still entered into the change registry as an emergency modification. I can't stress enough how important building and maintaining a robust change control framework is; not only will it help drive security, your entire IT infrastructure will benefit.

Wherever possible, use this audit phase to help build or validate your configuration management database/asset tracking process. When you encounter a system, you should be able to identify what OS/apps/data are on this box, who is the point of contact, and ideally who the data custodian for the box is. This will drive not only security and business continuity processes, and improve efficiencies for the IT group as well.

5. Validate Log Flows
Make sure you routinely validate that traffic from all devices is still inbound. Often, outages of a significant period get silenced — only to not be turned back on when the outage is resolved. I've seen a company go well over a year before realizing there were no alerts triggering in their SIEM from an entire branch location.

6. Check Your Capacity
When establishing or reviewing your SIEM, be sure that your storage capacity is appropriate for your logs. A lot of solutions base their pricing on consumption — a cap on events per second, for example. Sometimes, companies decide to limit what they are logging in an event to keep pricing to a minimum. This can be a crucial mistake. Many — probably most — penetrations of networks initially occur on individual user machines rather than on critical systems. If you are monitoring only critical or major systems to save money, you might very well be missing all the initial warning signals of the initial intrusion. Your first indication might end up being when you see your database being sent across the wire.

Proper capacity doesn't just apply to the size of the SIEM's hard drives. It is critical to ensure that you have adequate staffing for deploying and maintaining a SIEM. Don't underestimate the manpower required when you budget for operations.

A properly configured and maintained SIEM/IDS can help save your bacon. But as you've seen here, they aren't devices you can simply deploy and forget about. They can be a big financial and logistical burden, but if you follow the tips discussed above, they can also be your company's best friend.

Related Content:

Join Dark Reading LIVE for two cybersecurity summits at Interop 2019. Learn from the industry's most knowledgeable IT security experts. Check out the Interop agenda here.