Logging Isn't Hard -- Getting Started Is

Having performed numerous incident response engagements for small to midsize businesses, the thing that causes never-ending frustration is the lack of logging. There have been a few exceptions where a sysadmin had the foresight to log everything, but more often the IT staff is stretched so thinly that they can't implement something like logging, which doesn't immediately fix some ongoing problem. Instead, they're forced to fight fires that never seem to stop popping up -- and I'm stuck with no more than 30 days of logs to work with.

When things really go south is when those fires IT is fighting are the result of a compromise. What could logs have done at that point? The Verizon Data Breach Incident Report includes some great statistics, one of which says that evidence of a breach was found in the logs in more than half of the cases. That might not seem significant until one realizes that had someone been monitoring those logs, the incident might have been detected early and possibly prevented, saving thousands to millions of dollars.

I've spoken a lot about the importance of regular log monitoring, but it's important to point out that logs have more value beyond preventative and detective capabilities afforded by daily log monitoring. Taking the time to collect the logs also provides a forensic trail that can be invaluable after an incident occurs. An incident responder can leverage the logs to answer the tough questions about how an attacker got in, what systems might have been touched after the initial breach, and whether sensitive data was exfiltrated. But the caveat is that logging has to be enabled, configured properly, and sent to a secure location.

Considering how ridiculously low-cost hard drive storage is, there's no reason why the smallest SMB can't set up a server with a 1- to 2-terabyte hard drive to serve as central collection point. The setup doesn't have to be elaborate: A Linux server running syslog or a Windows server running the free version of Splunk will suffice in the beginning. The most important thing is to start logging so the logs are there when they're needed.

One of the simplest yet most effective solutions I've seen in one environment was a single Linux server logging for approximately 60 desktops and servers. The Windows desktops and servers used Snare to send their logs via syslog to the logging server. The Linux and Mac systems used native syslog functionality to forward their logs to the central server. A firewall configuration on the server only allowed inbound logs and nothing outbound, so all management, including log analysis, was done at the server's console. It wasn't elaborate. It didn't allow for early detection. But, it did provide great value post-compromise during a forensic investigation.

The biggest hurdle to logging is starting. Once that's done, it's just a matter of working in a little time each day to review the logs to help catch problems before they become the stuff data breach headlines are made of.

John Sawyer is a Senior Security Analyst with InGuardians. The views and opinions expressed in this blog are his own and do not represent the views and opinions of his employer. He can be reached at [email protected]