The Right Stuff

5:13 PM -- I'm currently writing a report for one of my clients on a server compromise they suffered just over a week ago. As I write it, it reminds me of just how important it is to have a competent system administrator who understands the systems he or she is hired to manage.

The victim in this case was a Web server. It was the main server for my client, and was being managed by its third consecutive sysadmin, who also just happened to be new to Linux. The compromised server was running several outdated Web applications. Add an inexperienced sysadmin and numerous outdated Web applications, and you have a recipe for pwnage (Wikipedia: Pwn).

In addition to extensive IT knowledge, good troubleshooting skills and patience, a successful sysadmin has to be able to monitor what's going on. In the case of my client, monitoring systems didn't alert the sysadmin of the problem, but instead several of her clients who reported problems with their Websites, which were hosted by the compromised Web server. Other organizations also sent notifications that my client's server was attacking their servers. This is obviously not the way you'd want to be notified of a compromised server.

Having dealt with this particular client on and off for a number of years on other security issues, I know that they have very little budget for IT (which you may have also guessed since this was the third sysadmin). When the company president began looking for a new sysadmin, he wanted someone with Windows and Linux management experience. But a small budget meant a small advertised salary, so there was an equally small pool of applicants to choose from. And since the organization had many more Windows desktops than Linux servers, the president ended up choosing a very competent Windows sysadmin with no real-world Linux management experience.

My final report for my client will contain a couple of recommendations. If they like their new sysadmin and aren't confident in finding a suitable replacement, then they need to migrate their Linux servers to Windows to fit the competency of this sysadmin. Or for the cost of the sysadmin's salary, they could outsource their Web hosting and hire one or two help desk staffers to support the Windows desktops.

I don't know what the client will choose to do, but I hope that their resulting server downtime and my report will help them realize that they need a sysadmin with competency in all aspects of IT, especially those technologies critical to their business. Only time will tell.

– John H. Sawyer is a security geek on the IT Security Team at the University of Florida. He enjoys taking long war walks on the beach and riding pwnies. When he's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading