What to Do When Your Security's Breached

Well, it's finally happened. Despite all your efforts to stop both internal and external attackers, someone has penetrated your defenses and stolen or damaged your data.

You've got a full-blown security incident on your hands. What are you going to do about it?

If you've been smart, experts say, you'll already have a computer security incident response team -- and a plan -- in place. You'll even have tested the team and plan in some sort of live simulation.

But many companies are too resource-stretched -- or too small -- to have a full-blown, fully-tested incident response strategy. Others don't have top executives who are forward-thinking enough to approve the development of such a plan. What happens then? Is there an "in case of fire, break glass" answer?

Not exactly, experts say. But if you've discovered a potential security breach, there are some steps you can take to stop the threat and seal it off. Just as importantly, there's some advice on what not to do, so that you can protect the "crime scene" and preserve your chances of catching the crooks.

The following are six steps you should take if you encounter a possible security breach. Some experts recommend eight, some have 10, but these are the ones that most authorities agree on. Oh, and they also agree on this: You should have done the first three steps before you had the breach.

1. Assemble an incident response team.
Before you can do anything, you need to make sure you have the right people in the room. A security breach typically isn't handled only by the IT department -- it may involve legal experts, top executives, public relations people, and representatives from all of the business lines affected. If the breach involves insiders, you may need someone from human resources. If it involves money, you may need a financial officer.

One of the biggest mistakes companies make is allowing one "cowboy" to handle an incident alone, says Brian Contos, CTO of ArcSight and author of Enemy at the Water Cooler, a book that analyzes many real-life computer incidents. "There's always one person that steps up and thinks they can take it on, but critical incidents usually need the attention of the whole group."

Ultimately, the computer security incident response team (CSIRT) -- including external consultants or forensics experts -- should be selected prior to an event, experts say. And the teams may vary according to the nature of the incident. If you don't already have a team in place, choose it quick, and make sure all the stakeholders are there.

2. Assess the initial damage and the risk for more.
Your response to a security breach will be commensurate with the risks to your business. An external attack on your public Website might not hurt much if you're a local machine shop, but it can break your business if you're Amazon.com. Likewise, an insider attack on the personnel database carries a different type of impact than the theft of a customer database.

According to BackGrounD Software, a Canadian forensics firm that does security breach damage assessment, the costs of a breach should include not only the technical costs associated with finding and fixing the breach, but also loss of productivity and loss of business. You'll need a plan that not only outlines your strategy for recovering your systems, but that includes steps for recovering customers.

Indeed, brand loss can be the most damaging element in a customer-facing breach, experts say. In a study published earlier this month, Javelin Strategy and Research reported that 78 percent of consumers would stop shopping with a retailer that lost or compromised their data. (See Banks, Retailers Seek to Regain User Trust.)

Next Page: Page Two

3. Develop a notification plan.
Now that your team has a grip on the incident and its potential impact, you need to decide who to tell, and in what order. If a potential crime has been committed, law enforcement might be one of your first calls. If you are planning to bring in third-party consultants, such as security experts or a computer forensics firm, you should bring them in as early as possible.

Notifying employees may require some training, experts say. You'll have to tell them how their systems may be impacted, and what to do -- or not do -- to protect them. You'll have to tell them what they can and can't say to customers or others outside the company.

"When the breach comes from some hacker in Romania, this process is pretty simple," Contos observes. "But when you suspect it's your boss who's the leak, it's a lot more difficult. Who you tell and when you tell them can make a difference in whether you're able to quickly find and fix the problem."

Most states today have specific deadlines for informing customers and others who may be affected by the breach. In most cases, these laws allow up to 30 days for disclosure, so in most cases, you will have time to work on the problem before you make a public statement.

4. Begin remediating the problem.
Experts can never stress it enough: Never begin remediation until you fully understand the problem and its potential impact. Many experts say you shouldn't touch anything until a forensics team has been called in, lest you damage the evidence or make the problem worse.

But if the breach is actively hurting your business, this rule of thumb may not make sense. In some cases, you will want to unplug the servers or storage systems that are being infected or penetrated in order to limit the damage. In the case of an Internet-borne breach, turning off Port 80 might help.

"Disconnect your server(s) from the network," recommends BackGrounD Software. "If there is a potentially malicious code running, disconnect media devices as quickly as possible (i.e. disks, SAN, NAS). You never know how far the intruder has managed to get, so the faster you disconnect the equipment, the more of a chance you have to save your data."

After that, however, the steps you should take will depend on the skills you have in house. "For a small company, I would say hold on and wait for the experts," Contos says. "But a large enterprise could have people trained to handle this sort of incident. It's a skill set. You can learn it." Some companies, such as Target, maintain a fully-trained forensics staff that may even be offered to other companies, he observes.

Some problems don't immediately present themselves as security incidents, and IT staff may have already treated them with everyday troubleshooting measures. "It's not like there's a special beeper that goes off when it's security-related," Contos notes. "The key is that when you realize it's a security problem, you stop and consider what you're doing. Don't keep making configuration changes."

5. Document everything.
Lack of documentation can not only make it difficult to rebuild your systems after an incident -- it can also hurt your chances to make a case against an attacker in court, experts say. Throughout the assessment and remediation process, you should record everything, from how the incident was first detected to how the various members of the CSIRT team responded.

In some cases -- particularly in the case of attacks from outside the company -- the documentation will be done automatically, through log files from firewalls, IDS/IPS systems, and/or security information management tools. These tools should record the intrusion, the subsequent infections or downloads, and the configuration changes that were made to halt the attack.

Insider attacks are a different story. "In a lot of cases, an insider attack isn't detected initially through traditional security tools, but because someone noticed unusual or suspicious behavior," Contos says. In these cases, it's important to document the reports and log the behavior in addition to using monitoring tools to track computer activity, he advises.

Experts agree that documentation is one of the most frequently overlooked and difficult tasks to execute during a security incident. In many cases, the players don't have time to record what they're doing, or don't feel it's important. In the end, however, the documentation can be the key to reconstituting systems that have been rewired to stop an incident -- or to making a legal case against an intruder.

6. Develop a strategy for stopping the next attack.
Doing a post-mortem on a security incident -- or developing a plan for responding to the next one -- may seem like longer-term activities. But if one attacker finds a vulnerability, there's a good chance that he may have accomplices -- or that another attacker might find the same vulnerability.

It's not unusual for attacks to come in bunches, so it's important to permanently seal off your leaks and decide how you will alter your response process if an incident occurs again, experts say.

"Review what you've done and decide how you could have done it better," Contos says. "More importantly, consider all of the subsequent risks and, if you don't already have plans in place, get them in place. If you had trouble getting top management to listen before, you won't now."

— Tim Wilson, Site Editor, Dark Reading