Realistic Patch Management Tips, Post-SolarWinds

Figure 1: (image by Barbara Helgason, via Adobe Stock)

"If you didn't already know that patching introduces risk, well...now you know," says Brad Causey, CEO of security consulting and penetration testing firm Zero Day Consulting.

Causey is referring of course to the recent attack on SolarWinds that shook the industry. Software updates for SolarWinds' Orion network management software were used to distribute the Sunburst/Solarigate backdoor Trojan to some 18,000 organizations worldwide. (Note: SolarWinds is, itself, also a provider of third-party patch management services. However, those services do not appear to have been affected by the recent attacks.) 

"We're introducing risk by trying to reduce risk," Causey says.

This isn't a new thing though, he says, and testing patches before deployment is standard best practice. Yet, patch testing is generally done to avoid operational snafus, not security threats. It's meant to spot a code library change that prevents three other applications from running; not to spot a backdoor Trojan.  

With the Sunburst/Solarigate attacks fresh in mind, though, is it time to revamp patch testing procedures? How can enterprise infosec teams tackle patch management securely? Here's advice from security experts on what to do now.

Be Realistic.

Causey and others say that a supply chain attack on the scale and sophistication of SolarWinds is harrowing, but it doesn't mean that enterprises need to completely reinvent patch management. Rather, IT teams just need to do some of the best practices they should have been doing all along. After all, the National Institution of Standards and Technology (NIST) lays out highly detailed guidelines on patch management in SP 800-40. 

The trouble is, SP 800-40 was last updated eight years ago, and by NIST's own reckoning, the number of vulnerabilities per year has tripled since then. 

"We patch all the time. We're always patching," says John Pironti, president of cybersecurity and risk consultancy IP Architects LLC.

Security hygiene, including patching, is an essential part of defense, says Pironti. Nevertheless, he says, "We're fooling ourselves if we think we can defend ourselves against a nation-state attack [like the SolarWinds incident] while continuing to release code at the speed we do."

Curtis Franklin, senior analyst of enterprise security management at Omdia, says companies must have patch management technology to help automate the process now, "because it's gotten really beyond human-scale at this point." 

Keep Trusting Patches. But... 

Despite the recent high-profile example of a malicious software update, Pironti says companies should not shy away from deploying updates.

"I think we would be doing ourselves a disservice if we started distrusting patches," he says. "I'd rather trust my vendors than question them when there's an exploit in the wild."

He does, however, say it's fair to ask for better security hygiene in the software development lifecycle. 

...Ask Software to Be More Trustworthy

"We've been trained as a society to accept flawed code," says Pironti. While regulations mandate that some industries' products meet certain safety and quality standards, enterprise software is largely unregulated. Pironti thinks that at some point this may change. "You can't let [software companies] be the barometers of what's acceptable and unacceptable risk," he says.

In the meantime, he suggests companies ask software vendors and service providers one question before any purchase: What are you doing to ensure the integrity of your third-party code?

Create a Testing Environment That's a Reasonable Representation of Reality.

In an ideal world, your testing environment would be a perfect mirror image of your production environment. It would represent every device, running every version of every operating system version, and every application, in every complex configuration that might be running in your environment at the time.

"And you'd have to invest in all that equipment nobody's using, and pay someone to maintain it, right?" points out Causey.

More realistic and affordable, though, he says, is to create a testing environment that accurately represents the systems that are the most critical -- those that are used by the widest number of users, or most critical to daily operations, or that touch the most sensitive data.  

Omdia's Franklin says that many companies succeed in creating a test environment that represents the lion's share of their endpoints. "The trouble is with their edge cases," he says.

Those edge cases might not be a problem. Until they are.

Franklin lays out an example:

It might be the system that prints out the bills of lading for the trucks leaving your manufacturing facility. And it runs a dot-matrix printer that has been cranking along since 1997. And Charlie at the freight yard knows how to hit the buttons on his Windows 98 computer to make it print all these bills of lading to keep things flowing out. And you've decided that it's simply impossible to retrain Charlie down in shipping. So you're not going to. 

And it was fine when Charlie was getting hand-written notes and typing them in. But sometime a few years ago your SAP rep said 'ya know we can put a connector that goes from SAP to Charlie's desktop.' So now Charlie's Windows 98 desktop has a link back -- probably through the Internet -- to your SAP instance.

Now, all of a sudden, Charlie's Windows 98 machine is a vulnerability. ... My guess is you don't have a Windows 98 machine in your [testing] lab. So even if [Microsoft] released an out-of-band patch for Windows 98, you couldn't test it.

You're going to have some cases like that. And they get far more numerous and bizarre in healthcare.

Franklin says most companies widely use sandboxing. "But if they're honest with themselves, they know that they can't sandbox everything. If they're doing it right, though they know what they can't sandbox." 

Use the Right Tool for the Job.

Identifying those fringe cases requires help. There are many types of technologies that will allow enterprises to locate and organize those IT assets and a wide variety of tools that help make patch management smoother. For example:

{Continued on Next Page}

{Continued from Previous Page}

Franklin says many products on the market will ably perform several of these functions.

But don't assume that all these functions are essentially the same thing. "Vulnerability management systems are notoriously bad at being asset-management systems," for example, Causey cautions. 

Patch management service providers like SolarWinds and ManageEngine vary in many ways. Some questions to consider when selecting one:

Isolate your Testing Environment. 

The Sunburst/Solarigate backdoor was delivered through a software update that had been signed by a trusted vendor; more payloads were then delivered via the backdoor.

Isolating the testing environment protects the live environment from such attacks. In the case of Sunburst, however, it might have eluded notice anyway, since it waited two weeks before making contact with its command server.

In an ideal world, Causey says, testing environments would be kept on their own physical servers. In a realistic situation, though, virtualization and microsegmentation are more feasible options. 

Back Up Before Deployment. 

Romanus Raymond of patch management provider ManageEngine says his firm always advises customers to create versioned backups before updating software, and to conduct those deployments to small groups at a time. This makes it easier to tell if a problem is due to a troublesome update, and gives the organization a recent, safe, known-good version to fall back on.

"We always recommend fragmentation," says Raymond, "so you can easily diagnose problems."

Prioritize Important Patches. But Don't Ignore Any. 

Start with high-priority patches, Causey says.

What qualifies as "high-priority?" Vulnerabilities with known exploits in the wild are on the list. He also advises that organizations pay heed to their threat intelligence and monitoring tools - these will help SOC teams determine which bugs are most critical to their particular environment, regardless of severity rating.  

Certain updates will have to wait patiently, but don't ignore them forever, says Pironti. Some software vendors will not provide support until you update software to a certain level, he says. 

Some Vulns May Never Be Patched. Act Accordingly.

Sometimes the risk assessment will show that the risk of deploying a patch outweighs the risk of the vulnerability itself. This is particularly prevalent in the case of industrial control systems and OT environments, but it may also be the case on pieces of critical enterprise IT infrastructure.  

Franklin says that companies will decide how to handle those cases: "'We know that we're not patching this. We have read the CVE. We know what the CVSS is. So we will bulk up our security infrastructure to try and make sure that that piece of kit never sees the threat that this patch is supposed to guard against.'"