Buggy CrowdStrike EDR Update Crashes Windows Systems Worldwide

This is a breaking news story and will be updated as new developments occur.

This morning, Microsoft servers across the world displayed the dreaded "blue screen of death," leading to mass IT outages that disrupted business, airlines and flights, healthcare providers, banks, and more. The cause: A defective update to CrowdStrike Falcon Sensor, a widely used cloud-based endpoint detection and prevention (EDR) software program.

CrowdStrike said its engineering team has identified the issue that caused the massive disruption to Windows-based systems: A bug in the Memory Scanning prevention policy, which was not identified during their testing stages, Callie Guenther, senior manager at Critical Start, noted in an emailed statement.

"While CrowdStrike likely performed standard regression and functionality tests, these were insufficient because they did not simulate the real-world deployment environment where the bug caused the Falcon sensor to consume 100% of a CPU core," she wrote. This ultimately led to system performance issues.

CrowdStrike has since reverted the flawed Falcon software update. Even so, some users are still experiencing system crashes or are unable to stay online to receive the new and fixed version. The cybersecurity vendor has provided workaround steps for this issue.

In a post on social platform X, Microsoft CEO Satya Nadella said the company is aware of the issue and is working closely with CrowdStrike to provide technical support to its customers and get their systems back online.

Microsoft 365's mitigation process is complete, and its telemetry indicates that all affected Microsoft 365 apps and services have recovered as it enters a monitoring period to ensure that its systems are fully resolved.

It does not believe that this outage is related to the "July 18 Azure outage that impacted a subset of Azure customers," stated a Microsoft spokesperson. "That issue has fully recovered."

Falcon Fallout

The severity of the broken CrowdStrike update became increasingly painful as victim reports rolled in throughout the day: More than 1,300 flights have been canceled or delayed, trains, card payments in stores, pharmacies, and even general practitioner (GP) surgeries were stalled. 

The Department of Health in Belfast reported that two-thirds of GP practices in Northern Ireland have been affected, with patient records inaccessible as well as lab tests and routine prescriptions.

Delta flights have been paused as it "works through a vendor technology issue," the New York Times reported, and Turkish Airlines has canceled at least 84 flights. Employees at financial institutions like JPMorgan Chase and Instinet have had trouble accessing their corporate systems as operations began to stutter.

The outage has also impacted Maricopa County Elections at certain voting locations. Voters are encouraged to visit Locations.Maricopa.Vote for up-to-date information regarding different voting locations.

Even the Paris Olympics organizing committee reports that its IT operations have been affected, mainly affecting delivery of uniforms and accreditations.

Meanwhile, President Joe Biden has been briefed on the outage, according to the White House, and administration officials are reportedly in touch with affected entities as well as CrowdStrike, which is working with customers that have been impacted.

"Mac and Linux hosts are not impacted," George Kurtz, president and CEO of CrowdStrike, wrote online. "This is not a security incident or cyberattack. The issue has been identified [and isolated,] and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website."

CISA stated in an alert that it is aware of the outage and has observed threat actors trying to take advantage of the incident via phishing and other malicious cyber activity.

"CISA urges organizations and individuals to remain vigilant and only follow instructions from legitimate sources," it stated in the press release. "CISA recommends organizations to remind their employees to avoid clicking on phishing emails or suspicious links."

It's Not a Data Breach, but it's a Disaster

In an industry where cybersecurity practices and services are meant to protect an enterprise without interrupting them, this outage proves that "even non-malicious cybersecurity failures can bring businesses to their knees," according to Maxine Holt, cybersecurity analyst at Omdia.

This massive incident underscores an over-reliance on cloud services, Holt noted in an online statement, and the outage may prompt organizations to reconsider moving their mission-critical applications to the cloud.

"Omdia's Cloud and Data Center analysts have long warned about over-reliance on cloud services," Holt said. "Today's outages will make enterprises rethink moving mission-critical applications off-premises. The ripple effect is massive, hitting CrowdStrike, Microsoft, AWS, Azure, Google, and beyond. CrowdStrike's shares have plummeted by more than 20% in unofficial pre-market trading in the US, translating to a staggering $16 billion loss in value."

As CrowdStrike will undoubtedly face scrutiny as it gets back on its feet, only time will tell how this outage could affect regulation and pressure on software vendors.

"We need stronger regulations and guidance on vendor responsibilities for functional testing," Josh Thorngren, security strategist at ForAllSecure, wrote in an emailed statement. "If you're not testing the behavior of your application under-expected (and unexpected) conditions with every update — this type of issue will always be a risk."