NSS Labs Revises Endpoint Security Test Model

[3/5/2020 This story was updated with a correction that NSS Labs will continue to use its "Recommended," "Neutral," and "Caution" notices with endpoint products, in addition to the new ratings system, and with additional detail that its nonprofit will cover consumer security products, not just IoT.]

Cybersecurity testing company NSS Labs, which was quietly acquired by a private equity firm late last fall, has launched both a new ratings system for endpoint security product testing for that product category, and a new nonprofit testing organization for consumers for security and Internet of Things (IoT) products.

NSS Labs in October 2019 was purchased for an undisclosed figure by private equity firm Consecutive Inc., a move that was not publicly announced by the companies but which they later confirmed. Multiple sources close to NSS Labs described the merger as a fire sale of sorts to restructure the company amid financial woes, but NSS Labs CEO Jason Brvenik tells Dark Reading that the deal represents a reorganization by the company in order to better focus its resources.

According to Brvenik, the previous venture capital (VC) model was not a fit for NSS Labs or the testing market, mainly due to VC focus on growth and product. NSS Labs was under pressure from investors to sell a security-as-a-service threat intelligence offering for exploits, but the now-defunct Cyber Advanced Warning System (CAWS) service failed to gather steam among enterprises. CAWS, which was developed by NSS and had integrated with various threat intel vendors in that space, alerted customers on active exploits in the wild. NSS Labs since has folded its CAWS technology into its testing as a bundled service offering, according to the company.

"What we heard from the market was they didn't want more work from us [with the service]; they wanted answers and not data that makes them do more work," Brvenik says.

"We're now back to focusing on what we are really good at and what we're known for," he says. "It allowed us to look more at what we deliver to market and to make pivots to the cloud" and other areas, he noted.

NSS Labs announced the new initiatives of new test rankings and a new nonprofit testing arm for consumer security and IoT products during the RSA Conference in San Francisco last week. The new product ratings method, which the testing firm has first launched for endpoint protection products, rates vendor tools based on the criteria of management, false-positive rate, resistance to evasion, total cost of ownership, and their block rate of malware, exploits, and targeted attacks. NSS said it will also will continue to flag products as "Recommended," "Neutral," or "Caution," as well as now rate the products on a grading scale of AAA as the highest to D as the lowest.

The Testing Conundrum
The new moves by NSS Labs come at a time when traditional security product testing is undergoing a slow but welcome transformation. Vendors and test labs long have had an uneasy and often contentious relationship over control of the testing parameters and process, and NSS Labs at times has been at the center of that battle: The company in May 2019 retracted and apologized for a 2017 publicly released endpoint protection test report on CrowdStrike's Falcon, which CrowdStrike in turn challenged in a lawsuit alleging that the test was incomplete and used illegally obtained Falcon software.

CrowdStrike had hired NSS Labs the year before to conduct private testing of Falcon, but later terminated the testing engagement over concerns over the quality of the tests after it detected legitimate apps as malicious. NSS Labs continued to publicly test Falcon, using software it had acquired through a reseller.

In September 2018, NSS Labs filed an antitrust lawsuit against cybersecurity vendors CrowdStrike, ESET, and Symantec, as well as the nonprofit Anti-Malware Testing Standards Organization (AMTSO), over a vendor-backed testing protocol it deemed as unfair and vendor-centric. AMTSO's testing protocol aims for transparency between testers and vendors.

But NSS Labs dropped the lawsuit in December 2019, citing "progress" in how AMTSO and vendors were working with test labs. That doesn't mean that NSS Labs is now all-in for the AMTSO testing protocol, however: Brvenik says NSS Labs has no plans to adopt the AMTSO protocol for its testing programs. "We have not seen sufficient evolution there," he says. "It remains a vendor-driven environment."

Meanwhile, enterprises — at which testing is aimed — have been caught in the middle of such spats and faced with an often opaque testing model that critics have described as a vendor pay-to-play. Most don't have the resources to conduct their own in-house testing of security products, so they are left with recommendations from consulting firms, third-party testing organizations, or just claims of the vendors.

Brian Monkman — executive director of NetSecOPEN, an industry organization that coordinates network security performance testing based on its Internet Engineering Task Force standard-based process — says enterprises should be able to get open and transparent security testing from a neutral third-party testing organization.

"When enterprises are looking at testing results to help them decide what security products get short listed, they need to look at how the testing was done and the level of detail, and what level of detail security product vendors are prepared to provide," Monkman says. "An open and transparent nature is starting to emerge in the endpoint testing market."

Take Mitre's commercial testing of endpoint security products, which it launched in late 2018. The nonprofit evaluates the products against its ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) model, using well-documented attack methods and techniques employed by nation-state and other advanced threat groups. Mitre's tests are based on open standards and methods, and the vendors perform live defenses with their products. It's a more collaborative test environment, many security vendors are embracing it, and the results are made public.

Meanwhile, the goal of NSS Labs' new product ratings scale is to ensure that organizations can choose the product or technology that best fits their needs, which may not be the most leading-edge product, notes Brvenik. Scoring endpoint products with a percentage grade is not necessarily representative of just how good a product is, he says.

"If you have five products and four of them are at 99.9% and one is at 99.5%, it's going to look like it [stinks] in a 2D [two-dimensional] axis, even though it's a great product. That model didn't fit well in that space," he says.

Chester Wisniewski, a principal research scientist with security vendor Sophos, says customers are demanding more transparency from security vendors. But there are plenty of challenges with endpoint testing, including that vendors can block only the threats they know about, he notes. "There's no way to test nation-state stuff" with today's tests, he says.

The underlying issue is that more attacks today are launched by humans behind a keyboard using stolen credentials. "The human will just keep changing the malware until they get through," Wisniewski says, a scenario that's difficult to simulate in most test environments.

IoT
The details of NSS Labs' new nonprofit are still being ironed out, but the organization will use NSS Labs' test infrastructure to put consumer security and IoT products under the security microscope and publish the results for the public. One concern is the intersection between enterprise networks and their users when they go home to smart devices and their Wi-Fi networks.

NSS Labs isn't the first to take on consumer IoT security testing: There's the Cyber Independent Testing Lab (CITL), a nonprofit led by Peiter "Mudge" Zatko and Sarah Zatko, which recently teamed up with Consumer Reports on a digital standard for consumer privacy, for instance. "They are doing cool consumer stuff, and [looking to conduct] cybersecurity testing in a rigorous" environment, says security expert Bruce Schneier.

Schneier also points to a security and privacy consumer labeling project underway at Carnegie Mellon University's CyLab, which is building a prototype Privacy and Security Label akin to a nutrition label that could be affixed to an IoT product's box. The goal is to help inform consumers about an IoT product's security and privacy features — or lack thereof — including how data it collects is used and whether or how it requires authentication, for example.

Related Content:

Check out The Edge, Dark Reading's new section for features, threat data, and in-depth perspectives. Today's featured story: "How to Prevent an AWS Cloud Bucket Data Leak."