Digital Detectives: Making The Most Of Your Incident Response And Forensics Efforts

Download the entire January, 2011 issue of Dark Reading.

Security professionals are overworked, stressed, and don't have enough hours in the day to do their jobs. Why? Too often it's because they're reacting, fighting security fires as they happen rather than providing proactive protection.

Reactive security is a common pitfall caused by staff shortages, lack of skills, and an overall low level of preparedness. Security pros need to make sure they're executing the full security incident response cycle: Prepare for problems, identify incidents, fix the problem and restore systems to service, and then loop back to preparation to make sure the lessons learned are applied to head off future problems.

It's essential today because there are now so many targets inside a company. "Before, with your firewall up and servers secured, you were relatively safe," says Chris Cuevas, a senior security analyst at the consulting firm Secure Ideas, who spent 13 years as a system administrator. "Now, everything is a target, from your cellphone to your printer and your webcam."

Prepare For Success

Companies need to implement routine activities to identify security incidents. Your staff will need to do daily log reviews to look for anomalous activities, do IDS event reviews, and follow up on user reports of abnormal system behavior. Events that surface from these reviews may prove unimportant, but they could provide the first hint of a serious incident in progress.

Preparation also includes staff knowledge. Regularly reading security news sites and blogs--such as DarkReading.com, SANS Internet Storm Center, and Krebs On Security; Twitter; and mailing lists, such as the ones at SecLists.Org--is needed to keep security pros current on threats and the tools to counter them. Experience isn't a substitute for knowing the latest security trends.

For formal training, some great programs are available from companies such as Mandiant and the SANS Institute that can be used to supplement the knowledge of experienced staff and get new employees quickly up to speed. The courses start with the basics of incident response and delve into advanced forensic topics, including memory analysis and malware reverse engineering.

But training alone isn't enough. Staff must have a clear process for how to effectively respond to incidents and work together constructively. "Teamwork and experience are critical to the success of an incident-response program," says Don Weber, a senior consultant at security consulting firm InGuardians.

To do that, security teams also need the right tools in place. The basic premise behind incident response is to have as little impact on the system at issue as possible while still getting the data needed to identify and contain the problem.

The tools you choose to use for a specific incident will depend on where it occurs and the particular responder's experience and training with the tools. Just because one tool works great on Windows 2000 doesn't mean it will work on Windows 7, and you certainly don't want to throw an advanced forensics tool like EnCase into the hands of someone who hasn't been trained in forensics and how the tool should be used.

Tool selection efforts must be synchronized with training strategies. Most training courses include a basic methodology that's based on a set of tools that have been vetted in a range of incident-response situations and tested to determine their accuracy and impact on a system. Examples include tools from AccessData, Guidance Software, Mandiant, and Microsoft Sysinternals. Others can be found in the books and cheat sheets listed on the next page.

Volatile Data, Respond With Care

Just the act of collecting information from a live system can lose volatile data--such as memory contents, network connections, open files, and the list of running processes--so you must collect that data first. The order in which you run your tools typically follows the "order of volatility" described by Dan Farmer and Wietse Venema in their book Forensic Discovery.

The recently released Tr3Secure Volatile Data Collection Script is a great example of a tool that considers the order of volatility as it collects data from Windows systems. The script first creates a forensic copy of the contents of the physical memory, then collects volatile data such as processes, network information, users logged on, open files, clipboard contents, and system information--anything that would be lost if power to the system were turned off. It then collects nonvolatile data like software installed, users and groups, and computer devices.

One common, and problematic, response to a security incident is to go right to the system in question and start running commands--kind of like what we see on TV. That approach goes against the goal of treading lightly during the incident-response process.

In an enterprise environment, there are countless places that incident-related information can be found and analyzed without ever touching or even running a tool on the target system. That's an important point, because if an attack is still going on, attackers will likely notice response efforts and try to cover their tracks by clearing system logs, or worse.

Centralized log management and a security information and event management (SIEM) systems are ideal sources of data that can be analyzed without interacting with the system in question. Intrusion-detection systems, antivirus software, network firewalls, backup servers, and network flow data are a few others that provide insight into an incident by detecting downloaded malware, port scans by a suspect system, and attempts to exploit a network-based service.

Social Media Exposure

To get a better idea of how these types of systems figure into a response effort, let's look at a fictitious, but all too real, scenario where an employee's system is compromised after the person browsed a social networking site. It's a big risk: 75% of companies in InformationWeek's 2011 Strategic Security Survey (see chart below) cite malware from malicious links as a risk from social networks.

Suppose an employee browsing a social networking site is tricked into clicking a link from someone who appeared to be a friend. The link turns out to be to a malicious website that exploits a flaw in a popular document viewer. Malware is then installed on the person's computer, letting an attacker pilfer sensitive company data and probe the internal corporate network for additional information.

Using an SIEM, an incident responder can identify the social network the person visited because a company's Web proxy, which monitors employee Internet usage, logged the activity. The subsequent download of executable content and repeated requests to a website overseas indicate the user's workstation is infected and is phoning home for further instructions. System logs and network traffic show installation of several back doors that are accepting incoming network connections from other compromised systems.

In addition to SIEM, there are other centralized logging tools and anti-malware software on the user's workstation. Antivirus software might detect a few pieces of the malware that were installed but not catch all of them. Subsequent components of the malware may turn off the antivirus and install new services. Those are all useful clues that should show up in the central logs and can be analyzed--even a lack of logs can indicate a problem because the malware may have cleared them in order to cover its tracks.

The network layer also is rich with log data from an IDS, firewall, and the routers themselves that can export network flow data. Backup servers also hold clues in situations where attackers wipe a server hard disk or have access for an extended period and add or delete files.

You must look beyond the compromised system to see the big picture and find all the information you need. Always ask where else in your network is there information that could help in determining what's going on. Then pull that data together, and analyze and cross-reference it with resources on the Internet to further identify and contain the incident.

Digging Deeper

Enterprise incident response and forensic solutions have advanced considerably over the last few years, making these specialized tools an attractive addition to existing enterprise toolsets. Vendors including AccessData, F-Response, Guidance Software, and Mandiant have created tools that help with everything from information gathering to more advanced digital forensics. Combined with trained, experienced staff, these products can aid in identifying, containing, and eradicating security problems

At the most basic level, enterprise tools let response teams gather volatile and nonvolatile data from one or many workstations and servers remotely and simultaneously when necessary. Responders can quickly identify an incident and correlate data across systems to find all affected systems--an indispensable ability when faced with geographically dispersed locations.

In the scenario above where a system was infected with malware through malicious links in a social networking site, the SIEM may notify the security team that a system is infected and antivirus tried and failed to clean it. If the SIEM is integrated with an enterprise incident response tool, it would automatically collect data, or a security team member could run the task directly from the tool's management console.

Then the initial information is analyzed by a member of the response team to determine if the events constitute a security incident. Process and network connection information and memory analysis could turn up evidence of new services on high-numbered, infrequently used TCP ports and data being sent outside the company. Using this information, the response team would contain the incident, killing the associated processes, removing the services, and analyzing the system to see what information may have been stolen--all of which an enterprise incident-response tool can help do.

Analysis doesn't stop with the one host. Enterprise incident-response tools can use the URLs, process names, and service ports that indicated the original host was compromised to comb through memory, processes, network connections, and hard drives looking for indications that other systems have been compromised. Entire hard drives and removable flash drives can even be imaged remotely over the network if a deeper forensic analysis is necessary.

Get Ahead Of The Problems

Many security pros get a charge out of figuring out the who, what, where, when, and why of a compromise. But frequent investigations take a toll, and it's unhealthy for an organization to regularly have to face down security breach crises. Let's look at how the company in our example above could be more proactive--before a breach, or using the attack as a spur to get more prepared.

First off, it could implement a Web security gateway to enforce its policy against public social networking sites, to block the download of executables and files detected as malware, and to prevent employees from accessing known bad sites. An application whitelisting tool could be used to prevent unapproved executables, like those that are part of the malware, from being run. Patch management tools can be used to patch vulnerabilities before they're exploited.

The company also needs to speed up its response time; if it takes too long to respond, additional systems could be compromised. That might be a problem with its tools; centralized logging and an SIEM can provide faster access to data and should be implemented to aid responders in making quicker decisions based on data from many systems.

Staff might need better training. For example, an IT pro with little security experience might have dismissed the initial event after seeing that the antivirus system identified and deleted a few malicious files. Or maybe the person did investigate the antivirus alerts and ran a few tools to collect volatile data but failed to see anything malicious because the attacker used a rootkit to hide malicious files, processes, and network connections.

Hindsight analysis must be the final step in any incident-response process. It's here that lessons learned can be assessed and applied to change operational strategies and procedures to prevent repeat incidents and provide faster, more efficient response in the future.

Reflecting back on where failures occurred isn't always a welcome process, but it's an opportunity to evaluate what's working and what isn't, and drive through needed changes that otherwise might face too much resistance.

The goal should be to get better security controls in place to prevent incidents and create a stronger incident-response process, one that can head off problems and also handle them faster and more effectively if they do occur.

Where To Get Help

Incident response cheat sheets

Advice from information security expert Lenny Zeltser Checklist from the SANS Institute Windows memory acquisition and analysis MoonSols Windows Memory Toolkit Mandiant Memoryze Mandiant Redline Enterprise forensic and incident response AccessData Enterprise EnCase Enterprise F-Response Enterprise Books Windows Forensic Analysis DVD Toolkit, Second Edition, by Harlan Carvey Digital Forensics With Open Source Tools by Cory Altheide and Harlan Carvey Malware Analyst's Cookbook And DVD by Michael Hale Ligh et al. Free incident response and forensic training resources A practical exercise from ForensicKB Information and downloads from Digital Corpora Examination of a hacking case from the CFReDS Project Network forensics puzzle contest Hacker challenges

Sidebar: 6 Stages Of Incident Response

Incident response is cyclical, with each stage feeding the next. The first two stages, Preparation and Identification, are a constant part of daily activities as security pros stay abreast of threats and monitor systems and networks. The six stages outlined below are taught as part of SANS Institute's incident handling and forensic courses.

1 | Preparation The security team builds a strong foundation by getting necessary training, acquiring and learning to use tools, and developing policy.

2 | Identification This is where a security event is determined to be a genuine problem. Accurate and accessible data is a must. IT reviews intrusion-detection systems, security incident and event management systems, and host logs; security teams acquire additional data from system administrators and run incident-response tools when necessary.

3 | Containment The team performs technical triage to prevent additional systems from being compromised and further data taken. It implements firewall rules, checks backup system, and coordinates with the Internet service provider.

4 | Eradication IT removes malicious components from affected systems or rebuilds them using trusted media and backups.

5 | Recovery Systems are returned to service and monitored for signs of more attacker activity.

6 | Lessons Learned A team must review the security incident, identify its root cause, and assess the incident-handling process to determine what should be improved. It creates an executive summary of the incident and implements process changes. --John Sawyer

John Sawyer is a senior security analyst with InGuardians. Write to us at [email protected].