NSS Labs' Abrupt Shutdown Leaves Many Unanswered Questions

When NSS Labs CEO Jason Brvenik gathered employees on a conference call the afternoon of Thursday, Oct. 15, the news he delivered came as a complete shock: The security product testing firm would be going out of business that very day. No severance packages for employees, and the engineers who had recently been hired by NSS Labs were now suddenly out of a job. The only public announcement of the move was a short post on its website: "Due to Covid-related impacts, NSS Labs ceased operations on October 15th."

Former employees who spoke to Dark Reading on the condition of anonymity say the sudden announcement came with few details, and several executives at the firm were also blindsided by the news. NSS Labs had been quietly acquired by private equity firm Consecutive Inc. in the fall of 2019 amid signs of financial struggle, layoffs, and restructuring.

Efforts to reach Brvenik, several members of the NSS Labs executive team, and Consecutive's partners for this article were unsuccessful.

Some former NSS Labs employees and executives paint a picture of a company that had struggled before the private equity deal but for the most part appeared to be on a path to regaining its footing this past year. Because NSS Labs execs had closely held the company's financial information, however, these sources note that they did not have direct knowledge of its financial posture.

Vikram Phatak, former president and CEO of NSS Labs until 2018, when he stepped down after suffering a heart attack, says his understanding is that the company's closure was precipitated by some "internal strife" at Consecutive. "I don't know the details of it, but NSS Labs was a casualty of it," Phatak says.

"If we weren't in the middle of a pandemic, the company would be strong enough to stay open, even without Consecutive" funding, he adds.

NSS Labs founder Bob Walder, who sold the company to Phatak in 2007, says he doesn't believe the pandemic was the main cause of NSS Labs' demise. "That sudden shutdown is really weird," he says.

Security product testing can be performed remotely, he points out, so the work-from-home shift in the pandemic should not have significantly impeded the company's operations.

Walder says he first learned of the company's acquisition in 2019 by Consecutive in a shareholder letter. "As shareholders, we didn't find out until after the deal was done," he says, adding that his remaining $290,000 to $360,000 worth of shares were reduced to a paltry $4.03 after the acquisition.

One former NSS Labs employee says Consecutive's investors pulled their funding of NSS Labs after some of their own funding dried up. "They imploded, so then we imploded," the employee says.

In an interview earlier this year, Brvenik said the sale to Consecutive was a way to reorganize NSS Labs and refocus its resources. The previous, traditional venture-capital model wasn't a fit, he said, due to VC focus on product and growth.

At the time of Consecutive's purchase of NSS Labs, the testing firm had been under pressure from investors to sell a cloud-based security platform called Cyber Advanced Warning System (CAWS) that monitored systems and security tools such as next-generation gateways and intrusion prevention systems against active threats, and checked security controls for organizations. But CAWS was a tough sell for enterprises and never really took off as an enterprise offering.

Phatak says he takes responsibility for the fallout from the decision to bring in VCs to fund the continuous-testing product for enterprises. "We were getting feedback ... that to really build that product takes [a lot of] money," he recalls. "But that decision was a mistake. Having two businesses under one roof just didn't work: You have to either be a software or services business. Trying to be both wasn't the answer."

He says the CAWS technology was strong, but it would have been better to have spun it off under a separate entity. While the VCs did their job, he says, the company's focus "went off track" during that period.

Phatak says he was surprised about the abrupt shutdown of NSS Labs because the refocus on its roots of testing-as-a-service appeared to put the company "back on track."

But not all former NSS Labs employees agree that the company was running a tight ship or testing service since the Consecutive deal. According to one former employee, vendors were getting disillusioned with the service because they felt it wasn't transparent enough and was unreliable, a complaint echoed both privately and publicly by some security vendors. With NSS Labs' major revenue coming mostly from vendors, this placed financial pressure on the company, the source says.

Walder notes that while vendors had to pay for private tests of their products, the group tests that NSS Labs conducted were "on our dime," he says.

Testers vs. Vendors
Friction between security vendors and independent testing labs is nothing new. It's an uneasy relationship, mainly over control of the testing process and parameters. The underlying issue, of course, is that enterprises need objective information about how these products and services stand up to threats. And vendors want to ensure their products test well. But someone has to pay for testing, and that someone traditionally has been the vendors.

NSS Labs over the past few years had an increasingly contentious relationship with several major security vendors. In May 2019, it settled a lawsuit with CrowdStrike over test results in the security firm's Falcon endpoint security product. In a confidential settlement, NSS Labs retracted the results in the disputed test, calling them inaccurate and noting that the test "was incomplete and the product was not properly configured with prevention capabilities enabled."

The February 2017 advanced endpoint protection test report had graded Falcon poorly, and CrowdStrike in the lawsuit had argued that the testing was incomplete and conducted using illegally obtained Falcon software.

In another high-profile case, NSS Labs in September 2018 filed an antitrust lawsuit against CrowdStrike, ESET, and Symantec as well as the Anti-Malware Testing Standards Organization (AMTSO), over AMTSO's vendor-backed testing protocol. In that suit, NSS Labs alleged it had "suffered antitrust injury" from AMTSO's standard and adoption by testing organizations.

NSS Labs dropped that suit in December 2019. In a statement at the time, Brvenik said AMTSO had "made progress to be more fair and balanced in its structure, vendors have shown progress in working with testing organizations, and the market itself has had significant change and notable acquisition activity." 

Enter MITRE ATT&CK
There's no silver bullet for security product testing, experts say. But MITRE's endpoint security testing service based on its ATT&CK matrix so far has many experts — and vendors — feeling hopeful that there is a way to conduct some tests in an open and fair way. MITRE publishes the product evaluations publicly online, so it's also free to enterprise organizations. Vendors voluntarily participate in the tests.

"MITRE will dominate the testing efficacy space going forward. This will be beneficial to the cybersecurity industry," says Tom Kellermann, head of cybersecurity strategy for VMware Carbon Black.

But MITRE's service, now under the auspices of MITRE Engenuity, isn't like other traditional endpoint testing, which mostly focuses on detecting malware. It pits products against known and documented attack methods and techniques used by a specific APT group — via its ATT&CK model.

"We are trying provide a deeper understanding of how [security] products address attacks in our knowledge base," says Frank Duff, director of ATT&CK evaluations for MITRE. But "I don't think we are a one-stop shop" for product testing, he says, because there are other metrics MITRE doesn't test, such as performance degradation.

So far, MITRE has conducted a few endpoint security products tests, including one against ATP3, aka Gothic Panda, one against APT29, and another against Carbanak, aka FIN7. Its next test round expands beyond endpoint security tools to industrial control systems, pitting them against the infamous Triton attack. 

Meanwhile, Phatak says he believes there is still a need for another NSS Labs-type company to offer testing services. "I think there's an opportunity," he says. "The key is [getting] the trust of enterprises. A business model needs to focus on serving their needs."

However, that's not a model that fits well with a VC-backed business, he adds, given VC demands for growth. "The results for vendors have to be honest and fair, but if there's a focus on getting enterprises the information they need, someone will succeed" at this business, he says.

"At the end of the day, it's probably not a venture-backed business."