CrowdStrike Fails In Bid To Stop NSS Labs From Publishing Test Results At RSA

A security vendor’s failed attempt to prevent a report containing details on its product’s capabilities from being released at the RSA conference in San Francisco this week has focused attention on the need for more widely accepted testing and validation services for security technologies.

The report is from NSS Labs, a company that bills itself as an independent security product-testing firm, which also sells its own threat protection platform.  NSS recently conducted security tests on Advanced Endpoint Protection (AEP) products from 13 vendors and released the results of its testing at RSA this week.

Each of the products, according to NSS, was tested against multiple attack threat vectors for their resilience against evasion techniques, their tendency to generate false positives, and other metrics. The products were then graded for their overall security effectiveness and total cost of ownership per protected endpoint agent and assigned a rating of “Recommended,” “Security Recommended” or “Caution”.

Security vendor CrowdStrike, one of two vendors in the report to garner a “Caution” rating filed a lawsuit in federal court in Delaware last week seeking to prevent NSS from releasing the report at RSA. In a legal brief, CrowdStrike claimed that NSS’ analysis of its Falcon advanced endpoint protection technology was based on incomplete testing conducted via illegally obtained access to its software and against CrowdStrike’s specific instructions to stop.

The court dismissed CrowdStrike’s request for a temporary restraining order and preliminary injunction against NSS Labs on the grounds that the company had failed to show how it would suffer irreparable harm from the report becoming public. It will now decide whether NSS acted illegally when conducting its tests on CrowdStrike’s Falcon technology.

George Kurtz, president and CEO of CrowdStrike says his company’s dispute with NSS has to do with the incompetent manner in which the tests were initially conducted and the fact that the results are based on incorrect and materially incomplete information.

CrowdStrike first hired NSS last year to conduct private testing on Falcon but quickly put a halt to it over concerns about the competency of the testing and the methods used, Kurtz says. As one example he pointed to NSS flagging legitimate software such as Skype and Adobe as malicious, during testing.

Even after paying a total of $150,000 for two private tests on Falcon, CrowdStrike was left with no confidence about NSS’ abilities and decided not to participate in a subsequent public test of advance endpoint protection products by NSS.

NSS however, proceeded to conduct public tests on Falcon anyway, using access it obtained illegally via a reseller. When CrowdStrike discovered what was going on the company immediately pulled NSS’ access to Falcon. So any subsequent conclusion that NSS made about Falcon’s effectiveness were based on incomplete and materially incorrect information and without all product features being turned on, Kurtz says.

“This is not about trying to silence independent research,” Kurtz says. “We welcome open, fair, transparent and competent testing. We didn’t necessarily see it here. This isn’t the Consumer Reports of cybersecurity. It’s bad tests, bad data and bad results.”

The episode shows why it is important to have more standard processes and organizations for testing the effectiveness of security products, he says. While companies like NSS Labs purport to be independent and unbiased, he says they have a for-profit business model and make money selling test reports and their own security platforms. “Unfortunately, there is no Underwriters Laboratories or testing house for security products,” Kurtz says.

Vikram Phatak, CEO of NSS Labs expressed regret over CrowdStrike’s stance. “We obviously disagree and are disappointed with Crowdstrike's characterization of NSS,” he said in emailed comments.

Though Crowdstike's request for a temporary restraining Order and preliminary injunction were denied by the federal court, the lawsuit itself is still pending.  “So we are limited in what we can say,” Phatak said. “Whether or not it is their intent, their suit has the effect of keeping us from debating the facts publicly.”

Such disputes over product testing and validation are not unusual says Pete Lindstrom, an analyst with IDC. Still, it is slightly unsettling when companies get as aggressive as CrowdStrike did in its dispute with NSS Labs, he says.

Technology buyers generally understand the limitations of product testing and know how difficult it can be for testers to replicate actual real world conditions when assessing the effectives of security products, Lindstrom says.

“In my mind security vendors should be looking for ways to get more information about the efficacy of their products out into the market,” he notes. “Anyone confident in their solution, should be willing to do that even within the limitations of testing.”

As long as all the circumstances of the testing are described fully and vendors have a chance to rebut claims, users are well served, Lindstrom said. An enterprise buyer that is already interested in CrowdStrike for instance is unlikely to view the company negatively based on one report, especially if all details are fully available.

 “These reports are generally used for confirmation rather than negation,” he says.

Related Content: