How to Interpret the 2023 MITRE ATT&CK Evaluation Results

By George Tubin, Director of Product Strategy, Cynet

Thorough, independent tests are a vital resource for analyzing vendors' abilities to guard an organization against increasingly sophisticated threats. And perhaps no assessment is more widely trusted than the annual MITRE Engenuity ATT&CK Evaluations: Enterprise.

This testing is critical for evaluating vendors because it's virtually impossible to evaluate cybersecurity vendors based on their own performance claims. Along with vendor reference checks and proof of value (POV) evaluations — a live trial — the MITRE results add additional objective input to holistically assess cybersecurity vendors.

This article unpacks MITRE's methodology for testing security vendors against real-world threats, offers Cynet's interpretation of the 2023 MITRE ATT&CK Evaluations: Enterprise results, and identifies top takeaways emerging from our evaluation.

How Does MITRE Engenuity Test Vendors During the Evaluation?

The MITRE ATT&CK Evaluation is performed by MITRE Engenuity and tests endpoint protection products against a simulated attack sequence based on real-life approaches taken by well-known advanced persistent threat (APT) groups. The 2023 MITRE ATT&CK Evaluations: Enterprise tested 29 vendor solutions by emulating the attack sequences of Turla, a sophisticated Russia-based threat group known to have infected victims in over 45 countries.

An important caveat is that MITRE Engenuity does not rank or score vendor results. Instead, it publishes the raw test data along with some basic online comparison tools. Buyers can use that data to evaluate the vendors based on their organization's unique priorities and needs. The participating vendors' interpretations of the results are just that — their interpretations.

What Do the Results Mean?

That's a great question — one that a lot of people are asking themselves right now. The MITRE ATT&CK Evaluations: Enterprise results aren't presented in a format that many people are used to digesting (looking at you, magical graph with quadrants).

Many independent researchers declare "winners" to lighten the cognitive load of figuring out which vendors are the top performers. In MITRE's case, identifying the "best" vendor is subjective. Which, if you don't know what to look for, can feel like a hassle if you're already frustrated with trying to assess which security vendor is the right fit for your organization.

With these disclaimers issued, I'll review the results to compare and contrast how participating vendors performed against Turla.

How Do You Interpret the MITRE ATT&CK Results?

The most important measurements to consider when reviewing the participating vendors' results are overall visibility and detection quality. There are a lot of other ways to look at the MITRE results, but we consider these to be most indicative of a solution's ability to accurately and effectively detect threats.

Threat Visibility

The ability to detect threats is the fundamental measure of an endpoint protection solution. Detecting attack steps across the MITRE ATT&CK sequence is critical for protecting the organization. Missing any step can allow the attack to expand and ultimately lead to a breach or other detrimental outcome. The Turla attack sequence was executed over 19 steps, which were broken out into 143 substeps. Visibility is the number or fraction of detections out of 143 possible chances.

An important note: MITRE allows vendors to reconfigure their systems to attempt to detect threats they miss or to improve the information they supply for detection. When this happens, the report notes that the detection happened with configuration changes. In the real world, we don't have the luxury of missing detections and then reconfiguring our systems, so the more meaningful measure is detections without the configuration change modifier. When you review MITRE ATT&CK Evaluations outcomes from the vendor community, prioritize detection counts that do not include configuration changes.

Analytic Detections

Analytic detections identify the tactic (why an activity may be happening) or the technique (both why and how the technique is happening) associated with the detection. These details are not only very helpful for security analysts when investigating an alert but indicate real threats vs. false positive alerts.

Vendors may not have provided analytic information (tactic or technique) for each of the 143 steps deployed in the Turla attack sequence. Again, it's best to measure analytic information before any configuration changes were implemented.

Charting Visibility and Detection Quality

This analysis illustrates how well the solutions did in detecting threats and providing the context necessary to make the detections actionable. Missed detections are an invitation for a breach, while poor-quality detections create unnecessary work for security analysts or potentially cause the alert to be ignored, which again, is an invitation for a breach. This chart quickly shows where each vendor scored on the two most important measures.

Still Have Questions?

Understandable. Cynet is hosting a webinar on Wednesday, Sept. 20, to review the newly released results and share expert advice for cybersecurity leaders to interpret the results to find the vendor that best fits their organization's specific needs. Cynet CTO Aviad Hasnis and ISMG Senior Vice President, Editorial, Tom Field will also share more details on the MITRE ATT&CK tests and outcomes. Additionally, you can review our full analysis of the 2023 MITRE ATT&CK Evaluation results.

About the Author

George Tubin is Director of Product Strategy at Cynet and a recognized expert in cybercrime prevention and digital banking and payments security. He was previously Vice President of Marketing at Socure and Senior Research Director with the leading financial services research firm TowerGroup (acquired by Gartner) where he delivered thought leadership and insights to leading financial services institutions, technology providers, and consultancies on business strategies, technologies, cybersecurity, and identity and fraud management.