MITRE Releases Results of Evaluations of 21 Cybersecurity Products

PRESS RELEASE

McLean, VA, and Bedford, MA, April 21, 2020 — MITRE released the results of an independent set of evaluations of cybersecurity products from 21 vendors to help government and industry make better decisions to combat security threats and improve industry’s threat detection capabilities.

Using its ATT&CK® knowledge base, MITRE emulated the tactics and techniques of APT29, a group that cybersecurity analysts believe operates on behalf of the Russian government and compromised the Democratic National Committee starting in 2015. The evaluations, which were paid for by the vendors, include products from Bitdefender, Blackberry Cylance, Broadcom (Symantec), CrowdStrike, CyCraft, Cybereason, Elastic (Endgame), F-Secure, FireEye, GoSecure, HanSight, Kaspersky, Malwarebytes, McAfee, Microsoft, Palo Alto Networks, ReaQta, Secureworks, SentinelOne, and Trend Micro, VMware (Carbon Black).

“The ATT&CK Evaluations help the cybersecurity community by improving the security products that we rely upon and arming end users with objective insights into those product capabilities to detect known adversary behaviors,” said Jon Baker, MITRE department head for adversary emulation and orchestration.

MITRE developed and maintains the ATT&CK knowledge base, which is based on real world reporting of adversary tactics and techniques. ATT&CK is freely available, and is widely used by defenders in industry and government to find gaps in visibility, defensive tools, and processes as they evaluate and select options to improve their network defense.

MITRE previously evaluated products from Carbon Black, CrowdStrike, GoSecure, Endgame, Microsoft, RSA, SentinelOne, Cybereason, F-Secure, FireEye, McAfee, and Palo Alto against the threat posed by APT3, a Chinese group that analysts believe is currently focused on monitoring Hong Kong-based political targets, and began releasing those results in late 2018.

“We’ve seen a huge growth in participation from our initial evaluations based on APT3 to this round of evaluations because vendors have seen the value of this kind of testing,” said Frank Duff, ATT&CK Evaluations lead. “We bring a very collaborative approach to evaluations, by working with vendors who want to improve their products, which ultimately makes cyberspace safer for everyone.”

The ATT&CK Evaluations team chose emulating APT29 because it offered the chance to evaluate the cybersecurity products against an adversary that uses sophisticated implementations of techniques through custom malware and alternate execution methods, such as PowerShell and WMI.

The team also made changes to the way that it presented the results based on feedback on the APT3 evaluations from analysts, vendors, and end users. The ATT&CK Evaluations website now features a tool that enables users to select particular vendors and display a side-by-side comparison of how they detected each technique, as well as a data analysis tool to take a deeper look at how they handled those techniques.

The team has also released a Do It Yourself APT29 evaluation that leverages CALDERA, an automated red team system that MITRE developed using the ATT&CK knowledge base. This enables users who are intrigued by the evaluations to test security products in their own environments against the same adversary. This may be particularly useful for organizations that can’t afford to employ a red team, Duff said.

Vendor Perspectives

Israel Barak, chief information security officer (CISO), Cybereason: “The ATT&CK Evaluations have really solidified many of the concepts that Cybereason has placed at the core of our product development and helped drive our product management. We’ve seen great alignment between how we, together with MITRE, approached the concepts of detection, triage investigation and response, both in our recent and upcoming versions.”

Dustin Duran, general manager of security research, Microsoft: “We believe this open testing approach gives customers a more informed view of the dynamic threat landscape and of sophisticated attacks faced today. Microsoft actively tracks and protects against these advanced threats like APT29, emulated in this simulation, which is why we’re proud to have contributed tangible threat intelligence on adversary behavior to the MITRE community so defenders can better hunt for, protect against, and ultimately prevent these kinds of attacks.”

Vladimir Kuskov, head of advanced threat research and software classification, Kaspersky: “This unique technical assessment compares the capabilities of EDR solutions in response to real-world adversary activities, with an unprecedented level of attack detail and execution transparency. Such tests reveal the overall level of industry readiness to address advanced threats and the gaps that need to be closed. Participating in the test has become a valuable experience which we already use to further improve our products. We look forward to taking part in Round 3 which will focus on FIN7/Carbanak.”

Jarno Niemelä, principal researcher, F-Secure: “MITRE’s evaluation helped us to further improve our detection coverage and also improve user experience of our solutions. Prior to the evaluation, F-Secure’s detection and response solutions were improved with built-in MITRE ATT&CK® categorization to provide standardized descriptions about the techniques used in the attacks, and the evaluation generated plenty of additional ideas on how to improve our offering.”

Alberto Pelliccione, CEO, ReaQta: Participating in the MITRE evaluation has allowed us to stress-test ReaQta-Hive in a very complex scenario. From this independent assessment, we’ve been able to evaluate our visibility and coverage in an objective and well-defined framework/approach meaningfully. For future comparative purposes, we can now rely on the framework as an objective standard. This evaluation served as a sort of collaborative environment that helped us to focus on those features that are more important when investigating sophisticated attacks. We are constantly innovating and measuring our capabilities, so the evaluation learnings were valuable in understanding what to prioritize for our end-users.

Michael Sentonas, CTO, CrowdStrike: “CrowdStrike firmly believes that independent third-party testing is critical to the cybersecurity industry, as it provides customers with an unbiased view on the effectiveness of the solutions they rely on to stop breaches every minute of every day. MITRE’s deep expertise allows them to look beyond measuring how security solutions react to malware, focusing holistically on extended attacks, reflecting the types of sophisticated techniques we observe in the real-world every day. CrowdStrike is proud to partner with MITRE on this mission to measure and drive continuous improvements in the cyber security industry.”

Benson Wu, CEO, CyCraft: “We see the ATT&CK Evaluations as a great playing field leveler. Finally, there is a place for vendors to go head-to-head in a transparent way that is meaningful to buyers and the rest of the industry--a veritable blue-team cyber colosseum for leading products around the world to benchmark their true capabilities. End users are often overwhelmed with marketing buzzwords and frustrated in the dearth of concrete info when comparing products to avoid redundant, weak, or non-existent capabilities. With the MITRE evaluation and accompanying matrices, end users, vendors, buyers and the industry at large now have a lexicon and a map to best spot, detect, respond to every move, and communicate effectively when facing sophisticated attacks.”

Yi Zhou, vice president of product management, HanSight: “The whole evaluation process is systematic yet comprehensive to help us improve our product from different angles, including identifying the blind spots in detection to improve our SIEM correlation rule design, evaluating the effectiveness for different types of Windows event log data sources for threat detection, and leveraging the multi-dimensional scoring system to help evaluate the detection result in a more objective manner.”

About ATT&CK

ATT&CK® was created by MITRE’s internal research program from its own data and operations. ATT&CK is entirely based on published, open source threat information. Increasingly, ATT&CK is driven by contributions from external sources. Cybersecurity vendors may apply to participate in the next round of evaluations, which will feature the Carbanak and FIN7 threat groups as the emulated adversaries, via [email protected].

About MITRE

MITRE’s mission-driven teams are dedicated to solving problems for a safer world. Through our public-private partnerships and federally funded R&D centers, we work across government and in partnership with industry to tackle challenges to the safety, stability, and well-being of our nation.

www.mitre.org