Bringing Science to the Debate

Repeat after me: "We don't know."

We don't know if responsible disclosure is better or worse than full disclosure. We don't know if releasing proof of concept or exploit code helps us or the bad guys, more or less. We don't know how the silent majority of IT workers want us in the security community to handle these issues, nor if their opinion is self-destructive or imminently practical. When you get down to it, we don't really know a fracking thing.

Sure, we all have plenty of anecdotal evidence to support our personal positions. We can all cite cases of this or that vendor tirelessly defending its customers, or putting them at mortal risk based on their handling of some vulnerability. We all know someone that suffered real losses at the hands of the latest random Metasploit exploit module, and someone else who used it to close critical holes in their security defenses before the bad guys made it in. We all talk about Blaster, Code Red, and other past incidents like they have any relevance in today's world, which we all also admit has changed completely from a few years ago.

There’s a word for picking and choosing examples to support a pre-existing belief without any scientific basis. It's called religion.

If Dan Kaminsky's recent unorthodox disclosure of the DNS vulnerability has done nothing else, it's polarized the IT community into a dozen discrete corners of our never-ending disclosure debate. Rain Forrest Puppy may have first formalized responsible disclosure back in the late 1990s, but we're far from any industry consensus. As an analyst, my customers were on all sides -- users, vendors, and researchers -- and I rarely saw agreement in any single demographic, never mind between them.

In a previous column here at Dark Reading I called the disclosure debate dead and received a bit of flack over that line, but the truth is the debate just hadn't advanced in meaningful way. It was nothing more than the same tired arguments on all sides.

I propose that it's long past time we brought some current science into the game. It's time to move past anecdotal evidence or one-off cases into wider-ranging realm of epidemiological studies. It's time to ask the users what they want, while developing risk metrics to allow them to make informed decisions despite their personal opinions. We may not reach definitive conclusions, and even if we do, they probably won't last nor change the minds of the truly religious. But it's always better to seek more data than to dismiss it before we even see it.

To that end, I'd like to take a baby step, with a small poll we're hosting here on Dark Reading. I've seen a bunch of polls on various blogs over this DNS issue, but no one has asked people to respond based on their role in the industry: end user, researcher, or vendor. Rather than a broad disclosure opinion poll, we're focusing on a single issue inspired by recent events -- the release of public exploit/proof of concept code at the same time vulnerability is disclosed. While we know this kind of a poll isn't statistically valid, it's at least a start.

Please take a moment to participate in our quick poll here at this link.

I was personally critical of HD Moore for releasing Metasploit exploit code for the DNS vulnerability so quickly after patches were available. My personal opinion is that Metasploit is so widely used, and so easy to use, that it empowers attackers on a scale far beyond the inevitable one-off exploits being rapidly developed across the globe. The scales are different for Metasploit, and while we need those modules eventually for testing, releasing weaponized versions too quickly hurts us more than it helps us.

But that's my opinion. In a recent blog post Richard Bejtlich made a cogent argument for the release of these same modules. It helped him, hurt some other end user clients I've talked with, but none of us really knows what's best on a broad scale.

Andrew Jaquith, an analyst at The Yankee Group, agrees. "The debates about full versus responsible disclosure, proof-of-concept code, and attack/exploit frameworks are passionate. People argue their points of view with incredible conviction -- but without any empirical evidence one way or the other,” Jaquith says. “What we need are metrics that show the effect -- or not -- of PoC/exploit code on customers. Is it helping them detect problems and fix them? Or does it increase their exposure to attack? The debate needs move from philosophizing to facts, and from dogma to data."

A group of us, including Andrew, have decided to take action and start collecting data. This poll is just a small way you can contribute, and if you are interested in being more involved you can email me at [email protected]. We don't know exactly what we're doing, nor how, but we do know it's time to get organized and take action.

It's time for more science, and less religion.

— Rich Mogull is founder of Securosis LLC and a former security industry analyst for Gartner Inc. Special to Dark Reading.