Lessons Learned In San Francisco Hack Job

At one time or another, it happens in almost every IT shop: A handful of people, or even one person, has the sole responsibility for and knowledge of critical systems. So much so that if any one of them were hit by a bus, your IT department would be severely hampered. Virtually every IT manager recognizes this problem, but far fewer do anything about it. San Francisco is now learning its lesson the hard way.

Terry Childs, an IT administrator with the city of San Francisco, whom the city had tried to fire in recent months, is alleged to have created a private administrative account on systems within the city's FiberWAN project. According to prosecutors, he's keeping the password a secret. In so doing, he's locked everyone else out of the system's networking gear and landed himself in jail, with bail set at $5 million. The city is trying to regain access to its network and has brought in experts from Cisco to help break into the compromised systems. The details are murky but also not vital to learning from this painful incident. Simply put, a trusted employee is accused of using that trust to hose his employer.

Organizations must trust employees to act honestly and ethically while at the same time acknowledge that employees pose one of the greatest threats. Employees must have access to systems, and that means they can do damage. InformationWeek's 2008 Strategic Security study found that 53% of respondents considered authorized users/employees one of the greatest threats, and 48% of respondents considered all employees one of the greatest threats. While most employees are ethical, all it takes is one disgruntled person to cause you headache and misery.

No technology or process can fully stop an insider, especially a knowledgeable insider like Childs, from causing harm--but what can be managed, and what's shocking about this incident, is its apparent magnitude. Technology can help. Strong authentication systems could have made creating unique credentials more difficult, and change management products might have alerted someone to Childs' activities. But technology alone isn't the solution; good management plays a part, too.

IT TAKES A TEAM
Jonathan Feldman, director of IT for Asheville, N.C., says problems like these can crop up when the IT department relies on heroes versus using a team approach. He describes the IT hero as a lone wolf, a go-to person who is solely responsible for critical systems. IT heroes often don't like to give up control over their systems, Feldman points out. These employees pose a real risk to the organization because if they leave the job, are unable to work, or simply decide to strike back, no one else has the knowledge to work around them.

What's At Stake • NO ADMINS
Childs changed router and switch administrator passwords

• CITY's BACKBONE
The network serves 60% of the city's needs, including some police systems

• PAID TO SIT
Childs is suspended with pay while he sits in jail

The solution is to cross-train IT staff so that no one person alone understands and controls key systems; spread responsibility for systems over several people; and adopt a practical change management process. By doing those three things, the harm that any one person can inflict on your IT department is lessened because there are redundant skills and knowledge in play.

Building an IT team rather than relying on lone wizards requires management discipline and may seem a more costly approach; IT heroes are an easy shortcut for even the best-intentioned managers. However, the payoff in building a team is a more resilient IT department. "You know you are successful when you can let IT personnel go on vacation without having to ask for their cell phone and hotel numbers in case something comes up," says Feldman.

Rogue employees can still wreak havoc if they want. The best you can do is to take steps to minimize the damage.