Debugging Your Bug Software

3:52 PM -- There's been talk recently about malicious attackers compromising software while it is still in the development stage. (See Hackers Attack Apps While Still in Development.) But what precautions can you take to ensure that the software you're buying or installing hasn't already been compromised?

Open source tools make their source code available. Of course, if you aren't a programmer and don't have programmers on staff to analyze the source code, that isn't much help. Popular open source security tools like nmap and Metasploit have a few years of development under their belts, so you can safely assume that their source code is being analyzed by many eyes. But there is no guarantee.

Proprietary tools present an even more difficult problem, since the source code isn't available for analysis. It's possible to disassemble these tools to track down potentially malicious sections of the binary, but that's an exercise most security professionals can't do -- or don't want to try.

So what do you do to make sure security tools aren't causing you to become more insecure? The first step -- and the most futile -- is to submit the tool to Virus Total for analysis by numerous antivirus engines. This is futile for two reasons: Historically, antivirus companies have flagged many security tools as malicious because they are used by malicious attackers. Also, many tools are infected with custom backdoor code, and signature-based antivirus tools are unlikely to catch it.

Installing and monitoring security tools in a closed test environment is the best method for identifying potential problems. All modifications to a system should be monitored. There are numerous solutions for doing this in Windows and *nix, such as InstallWatch, InCtrl, filemon, regmon, tripwire, samhain, and aide.

Network traffic and additional opened ports also need to be monitored to check for backdoor or phone home activity. Native utilities, like netstat, and third party utilities (tcpview, Wireshark) are available in Windows and *nix to accomplish this.

While VMware, or a similar solution, may be the most convenient way to create a closed test environment, it is far from ideal for this particular purpose. The bad guys can instrument a tool to detect when it is being run inside a virtual machine. We've seen this happen in malware, so it's not out of the question that an attacker would include this in a trojaned security tool.

If the tool knows it is running within a virtual machine, it might behave differently, making detection difficult. For the paranoid, this means you're now going to need a separate physical machine on a closed network just for testing security tools.

A closed physical network may not be enough, though. Have you seen the location-aware endpoint protection solutions from Symantec, Senforce, and McAfee that change your system's security settings depending on your location? A malicious attacker could use the same techniques to determine if you're running the security tool on a closed network by seeing if certain Internet locations are accessible. If the checks fail, then the tool won't phone home with your machine's current IP.

There are so many variations of potential attacks that it's impossible to try to list them all here. The key point is that you don't always see what you get. There may be "hidden features" that jump out and bite you later on. Monitor your security tools once they're in production to just be sure.

Do your due diligence to make sure that you don't become the weak link in your company. There's nothing worse than being the security guy/gal who's responsible for the company network getting pwned.

– John H. Sawyer is a security geek on the IT Security Team at the University of Florida. He enjoys taking long war walks on the beach and riding pwnies. When he's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading