Tool Validation: Trust, But Verify

I received a lot of great feedback after my Friday post about WinFE, the bootable Windows Forensic Environment. The biggest question was whether it really is treating the drive as read-only. In my closing, I said I'd do more testing than just building the CD and making sure it booted up in my virtual machine environment. As security professionals and forensic investigators, don't you all validate your tools before using them?When I took my first official forensic class about four years ago, I remember one of the biggest responsibilities I faced as a forensic examiner was to validate my tools to make sure they were doing what they were supposed to be doing. Questions I was asked included, "Are you sure that hardware write-blocker is really preventing data from being modified on the drive? How can you verify this?"

Don't think that this is specific to forensic tools. The same need for validation exists for security tools, such as vulnerability scanners, intrusion detection systems, and data discovery tools. If you have a vulnerability scanner that is supposed to check for certain vulnerabilities, then set up some vulnerable systems to make sure it is doing its job properly.

You don't want a false negative to occur where you thought system was patched only to find out it wasn't because it was exploited. This is a good example of where a patch management program can be validated via vulnerability scanning.

Intrusion detection systems are in a similar boat. You don't want an overwhelming number of false positives, but you also don't want any false negatives. An example of how to test whether your IDS is functioning properly is to use a penetration testing tool, like Metasploit Framework or Core Impact, and exploit some systems to ensure the IDS is picking up the attacks properly.

Now that I think about it, I probably spend more time testing some tools than I do using them in a production environment. As Ronald Reagan was fond of saying, "Trust, but verify." That's how I feel about the tools I use. Hopefully (and before my next blog entry) I'll have some time to verify the integrity of WinFE and can report on my findings. Either way, you should do the same.

John H. Sawyer is a senior security engineer on the IT Security Team at the University of Florida. The views and opinions expressed in this blog are his own and do not represent the views and opinions of the UF IT Security Team or the University of Florida. When John's not fighting flaming, malware-infested machines or performing autopsies on blitzed boxes, he can usually be found hanging with his family, bouncing a baby on one knee and balancing a laptop on the other. Special to Dark Reading.