CrowdStrike Tries to Patch Things Up With Cybersecurity Industry

UPDATED

A combination of factors caused the CrowdStrike Falcon endpoint detection and prevention (EDR) sensor to crash, resulting in the global outage affecting 8.5 million Windows systems in July, the company said last week in a root-cause analysis of the incident. At the same time, CrowdStrike CEO and founder George Kurtz and president Michael Sentonas were at Black Hat in Las Vegas with a public mea culpa.

CrowdStrike documented in its root-cause analysis that a mismatch between inputs validated by a Content Validator and those provided to a Content Interpreter resulted in an out-of-bounds reach issue in the Content Interpreter. Tests during development and release did not uncover the issue.

"Sensors that received the new version of Channel File 291 carrying the problematic content were exposed to a latent out-of-bounds read issue in the Content Interpreter. At the next IPC notification from the operating system, the new IPC Template Instances were evaluated, specifying a comparison against the 21st input value. The Content Interpreter expected only 20 values," CrowdStrike said. "Therefore, the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash."

While CrowdStrike says this exact scenario will not recur, the company is making changes to its process and mitigating steps to "ensure further enhanced resilience," the company said. CrowdStrike has also engaged two software security vendors to conduct an extensive review of the Falcon sensor code for security and quality assurance, and an independent review of the end-to-end quality process from development to deployment is underway.

'Owning' Its Mistakes

At the Innovators & Investors Summit at the Black Hat USA conference in Las Vegas, Rain Capital general partner Chenxi Wang kicked off a panel she was moderating with a question for CrowdStrike's Kurtz: "What happened?" Kurtz apologized to the room  — an action that appeared to be well-received by the audience — and noted that the company had released the results of the root-cause analysis.

The company acknowledged its failures again a few days later, when Sentonas was on hand Saturday at the DEF CON hacker convention to accept the 2024 Pwnie Award for Most Epic Fail. The Pwnie Awards recognizes the most outstanding achievements as well as the greatest failures in cybersecurity over the past year. The Most Epic Fail category is for a "spectacularly epic fail — the kind of fail that lets the entire infosec industry down in its wake," according to the Pwnie Awards' description.

The massive global outage made CrowdStrike an automatic winner, the Pwnie Awards stated last month. The outage's global impact was highlighted by the fact that CrowdStrike was awarded a two-tiered trophy instead of the traditional small, pony-shaped trophies awarded to winners in other categories. Sentonas said the trophy will be displayed at the company headquarters in Austin, Texas, to serve as a reminder to staff that "these things can't happen."

"Definitely not the award to be proud of receiving," Sentonas said in his acceptance speech. "I think the team was surprised when I said straight away that I'd be coming to get it. We got this horribly wrong. We've said that a number of different times. It's super important to own it when you do things well. It's super important to own it when you do things horribly wrong, which we did in this case."

This story was updated Aug 12, 2024, to correct inaccurate reporting stating the out-of-bounds reach issue was separate from the input mismatch.