Fallout from Rushed Patching for Meltdown, Spectre

Intel's unusual advisory yesterday urging its customers and partners to refrain from applying some of its firmware patches for the so-called Meltdown and Spectre flaws in its microprocessors illustrated just how pressured patching can backfire.

Navin Shenoy, executive vice president and general manager of Intel's Data Center, in a post called for customers and OEMs to halt installation of patches for its Broadwell and Haswell microprocessors after widespread reports of spontaneous rebooting of systems affixed with the new patches. Intel now plans to issue a fix for the Meltdown-Spectre fix, according to the company.

It's the latest in a string of missteps in the wake of the major disclosure earlier this month of critical flaws in most modern microprocessors: a common method used for performance optimization could allow an attacker to read sensitive system memory, which could contain passwords, encryption keys, and emails, for example. The vulnerabilities affect CPUs from Intel, AMD, and ARM.

Microsoft also has experienced problems with its operating system patches that provide workarounds for the microprocessor vulnerabilities, specifically its updates for Windows 10 on AMD microprocessor platforms. The vendor yesterday came out with new updates that resolve booting issues the original patches had caused. That came after compatibility problems with antivirus programs running on Windows that hadn't been updated for the Meltdown and Spectre workarounds.

The recently discovered Meltdown and Spectre hardware vulnerabilities allow for so-called side-channel attacks. With Meltdown, sensitive information in the kernel memory is at risk of being accessed nefariously; with Spectre, a user application could read the kernel memory as well as that of another application. The end result: an attacker could read sensitive system memory containing passwords, encryption keys, and emails — and use that information to help craft a local attack.

Both Intel's and Microsoft's patching problems underscore the downside of applying patches under pressure. "We've been telling our clients 'don't panic patch,'" says Neil MacDonald, vice president and distinguished analyst at Gartner.

Organizations such as cloud providers and large server farm environments were among the first to install the Intel and other vendor patches because they were at higher risk. Cloud providers, for example, had obvious concerns about customers suffering attacks via their servers, MacDonald notes. But some early adopters "got burned" with Microsoft's antivirus incompatibilities and locked AMD machines with the Windows patches, and unexpected reboots from the new Intel patches, he says.

Most enterprises can afford to hold off on fully patching for Meltdown and Spectre for now until the patches are fully vetted, however. The good news is there are no known attacks in the wild, which allows for a more risk-based rollout of patches, he notes.

"People who rushed to patch are getting penalized," MacDonald says.

Gartner is advising its clients to prioritize the systems they patch. If performance penalties with the updates are one of the side effects, then in some cases it's best not to patch at all, or to just apply operating system and browser patches. For some endpoints, for example, it makes more sense to patch the OS now and then the firmware later. "You'll get at least partial protection," MacDonald says.

Servers should be locked down, too, to mitigate the attacks. "They should not [be able] to execute arbitrary code, or do email … so servers should go to whitelisting," which would provide "significant" protection from a Spectre or Meltdown attack, he says.

Some systems may not merit patching at all, such as I/O-heavy network appliances, storage appliances, and security appliances, where the Meltdown/Spectre code update's performance hit would be detrimental. "In some cases, the appropriate risk-based decision is not to apply the patch because of performance implications," MacDonald says.

The performance hit with the patches is especially painful for the industrial environment, which is both a juicy target for attack as well as highly disruption-averse. "In the world of critical infrastructure, where safety and availability are paramount, updates that carry this kind of baggage are simply not applied immediately," says Eddie Habibi, founder and CEO of PAS Global. "The first option for facilities right now is to validate existing security controls and consider adding new ones only where risk is perceived as outsized."

Intel, Microsoft, Linux, and browser vendors' security updates and patches for Meltdown and Spectre are mainly workarounds and mitigations. A real fix requires a brand-new generation of microprocessors, a development that realistically is a year or two away at best, Gartner's MacDonald says. "There is no easy fix. These [patches] are all workarounds until new hardware is released."

Intel's patch glitches are due to its rushing them out without fully testing them for a cloud provider's environment of millions of servers, for example, he notes.

Meantime, Linux creator Linus Torvalds isn't happy with Intel's approach to working around the design flaw. In a post on the Linux Kernel Mailing List this week, he unleashed his frustration with Intel's workaround, calling it "garbage."

Related Content: