Rapid Triage To Stop The Data Bleed

The SANS Internet Storm Center on Tuesday questioned whether an exploit was out in the wild for MS09-039 due to increased scanning for TCP port 42. That same afternoon, a notice went out to the EDUCAUSE Security mailing list with the subject: "CRITICAL: Active exploitation of MS09-039 in the EDU sector." It's not often we get to see a preauthentication attack against a Windows service like WINS that makes an easy jumping-off point to compromise an entire Microsoft Active Directory. Can you imagine all systems being compromised in your company?The e-mail notification went further, saying that the vulnerabilities were being actively exploited and several institutions had been compromised -- "one of which also involved compromise of a small AD [Active Directory] domain." Can you fathom dealing with a compromise that involved everything? I''ve imagined the scenarios, and have ideas of how I would approach the situation. But I've never had to put it into action at such a grand scale.

Do you build a parallel network and slowly build a new domain by rebuilding machines and moving them over to the new network? Do you take everything down and rebuild offline? Or do you try to clean and rebuild in place, while you're still not sure what is compromised? (In case you're wondering, retirement isn't an option, but you might want to update your resume).

Michael Murphy and Aaron LeMasters spoke at Black Hat USA on this very topic in their "Rapid Enterprise Triaging (RETRI): How to run a compromised network and keep your data safe" session. In other words, when your network isn't your network anymore, this is how you protect your data and rebuild. They started off by saying the old incident-response process was broken and the RETRI approach focused on the containment phase.

Taking the idea of triage literally, RETRI contains the compromised network to prevent sensitive information from bleeding out. It's an expensive and involved approach because it does involve building an entire parallel network, but based on their case studies looking at rebuilding while online and offline, it was cheaper overall. And it prevents additional data loss while allowing employees to continue working.

RETRI won' work in every situation because it does require preparation -- some of which should already be in place for disaster recovery. And questions need to be answered. Are critical backups available for DNS, file servers, domain controllers, etc.? Is there evidence that a full rebuild is necessary?

There's a lot more to the RETRI approach than I can fit in here. One of the exciting things to come out of the presenters' work is a soon-to-be-released tool called Code Word. It is an enterprise incident response tool that will be released as open source. Based on the description, it sounds very similar to expensive enterprise IR tools from the big names like MANDIANT, AccessData, and Guidance Software.

Hopefully, you never have to experience an incident of the magnitude that would require such an involved and drastic approach as RETRI, but it's comforting (in a slightly disturbing way) that there are ways of dealing with it. The associated white paper and slides are available on the Black Hat Website here, and Code Word will be released here.

John H. Sawyer is a senior security engineer on the IT Security Team at the University of Florida. The views and opinions expressed in this blog are his own and do not represent the views and opinions of the UF IT Security Team or the University of Florida. When John's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading.