Companies Struggle to Recover From CrowdStrike's Crippling Falcon Update

A week after an ill-fated update from cybersecurity giant CrowdStrike knocked out an estimated 8.5 million Windows computers, causing problems ranging from downed medical systems at healthcare facilities to delayed flights for many airlines, organizations are still trying to restore access to their remaining affected systems.

Healthcare companies are among the most impacted organizations, with the corrupt file affecting about half of the members of the Health Information Sharing and Analysis Center (Health-ISAC), says chief security officer Errol Weiss. As of July 25, only 18% of affected organizations had fully recovered their systems, while three-quarters of companies still had as many as 25% of their systems still needing attention, Weiss says.

Many organizations had Windows-based medical devices, and now they are likely looking at a long-tailed recovery, says Weiss.

"My guess is that a lot of automated remediation was shared on Friday and Saturday — those methods probably helped a lot to get to the majority of completion," he says, referring to tools and scripts provided by Microsoft, CrowdStrike, and other companies. "But some of those scripts and automated fixes probably won't work on the kinds of devices that we're talking about, and now healthcare organizations have to take a manual look."

Microsoft released a USB Recovery Tool that gives administrators the option to use a USB drive to recover impacted systems from WinPE or safe mode. The tool may recover from safe mode even if BitLocker is enabled on a device and a recovery key is unavailable. There are also detailed recovery steps for Windows clients, servers, and operating systems hosted on Hyper-V, as well as affected Windows 365 Cloud PCs and Azure virtual machines.

Measuring Impact of the Outage

On July 25, CrowdStrike estimated that 97% of affected computers had returned to active status, as measured by the state of its Falcon software at the center of the outage. Managed security services provider Quest Software, whose customers span a gamut of sizes, is still offering aid to those working through the issue. The remaining companies likely represent a few hard-to-patch systems at larger firms and a large number of smaller firms that do not have the technical expertise to easily recover, says Kent Feid, senior director of product management at Quest Software.

"That 3% really represents the number of devices, and so that would equate to likely a substantial amount of small businesses still being impacted that are still somewhat unsure how to attack this," he says. "Smaller businesses tend to leverage more IT generalists or don't staff IT experts in-house."

The vast impact of the outage has still not been tallied, but insurance services firm Parametrix estimates that the event impacted a quarter of the Fortune 500 companies, with losses reaching $5.4 billion. That includes nearly $2 billion in losses for healthcare and more than $1.1 billion for the banking sector.

Even With Tools, Many Companies Worked the Weekend

While the recovery process is, for the most part, fairly simple, technical experts have gauged that each system requires an average of 15 minutes to recover because an administrator is required to physically access each system. In addition, companies that used BitLocker to encrypt the hard drive — a cybersecurity best practice, especially on laptop systems — would have to find the encryption key and input that key at the beginning of the process.

"There's no way to do this remotely because it has to be done in safe mode, where networking isn't working, so you can't connect to the machine remotely," says Vadim Vladimirskiy, CEO of Nerdio, a virtual desktop management firm.

At least 700 outages coincided with the bad update from CrowdStrike, with 39% of outages rated Critical. Source: Parametrix

Nerdio, which provides virtual desktops for its customers, said its customers were only minimally impacted by the failed update and its cloud desktop systems were easily repaired by recovering to a previous image. While many customers connect to Nerdio's service using a Windows computer, only the systems left on during the 78-minute window — during which the bad CrowdStrike update was distributed — were impacted. Affected customers could just switch to a different system to access their virtual desktops, limiting any impact, Vladimirskiy says.

Ironically, healthcare firms recovered by falling back on measures implemented to protect them from a threat CrowdStrike is deployed to prevent: ransomware. Health-ISAC's Weiss compiled a list of systems affected by the attack, and it includes patient services, lab collections, secure file transfers, dictation and transcription services, shipments, electronic medical records, and Medicaid and insurance billing.

"I started hearing about the impacts to these organizations and looking at the list, and it's like, 'My gosh, this sounds just like another ransomware incident,'" he says. "So that's what was going on inside healthcare on Friday for those organizations that were impacted by this. They just went, 'OK, systems are down, we're going to the manual backup procedures, we're going to paper,' and they knew what to do because they were drilling [their response to ransomware] in the past."

Preventing the Next Big Failure

The bad update also came after a significant outage of Azure services affected an above average number of companies, according to Parametrix. (On average, about 300 service disruptions occur among the Fortune 500 every day, the firm said. On Thursday, July 18, 419 coincided with the Azure outage, and on Friday, at least 700 occurred as the company dealt with the bad update from CrowdStrike.)

While CrowdStrike is feeling the market's wrath right now, the company is unlikely to be down for long because businesses need the type of services the company — and others like it — provide, says Quest Software's Feid.

"No software development company — I include ourselves in that — is perfect, right?" he says. "What's hard, I think, especially in the security industry and specifically for a company like CrowdStrike, they are being looked at and relied upon across a large part of the market to protect endpoints. ... The product is specifically designed to be ahead of the curve as much as it can, which means you can't have it both ways, consumers. There's always going to be inherent risk."