The SCADA Patch Problem

First in an occasional series on SCADA security

If you think database patching is onerous and fraught with risk, then try patching a SCADA system that's running a power plant. With post-Stuxnet paranoia pressuring major SCADA vendors like Siemens to regularly respond to vulnerability finds with software patches, utilities and other organizations running industrial control systems (ICS) face some serious decisions over where and when to patch -- if at all.

Many do not.

Vulnerability research is now looking more closely at SCADA products in the wake of the Stuxnet attack that shook the ICS world to the core by demonstrating how a little malware can go a long way to sabotaging a factory. Not all SCADA vendors are responding, but many of those who are find that their patches often fall on deaf ears.

Overall, only about 10 to 20 percent of organizations today actually install patches that their SCADA vendors are releasing, according to SCADA security experts. Responding to a Siemens or Rockwell Systems security update is not exactly Microsoft Patch Tuesday: Utilities and ICS organizations face risks of power shutdowns if a newly patched system goes awry. Patching workstations and servers is less dicey than a factory-floor or power-generation system, and experts say those systems are getting patched more regularly than plant-floor systems because they have shorter life spans and less direct impact on operations.

[More SCADA bugs, exploits in the wake of Stuxnet, but gradually improving security in some products, new data shows. See SCADA Security In A Post-Stuxnet World. ]

Decades-old process control equipment won't ever see a patch, anyway, experts say. Some plant equipment is so old that no one dares to disturb it. "I can tell you flat out that the people who run the equipment will not pursue patching aggressively. There are a lot of controllers out there from the 1960s and '70s that can't handle sophisticated security. I've dealt with a PLC [programmable logic controller] with bytes of memory -- you can't even put anything on there," says Andres Andreu, chief architect and vice president of engineering for Bayshore Networks. "To actually patch at that level is unrealistic ... There's legacy code written 30 years ago, and no one wants to touch that."

Eric Byres, CTO of Belden's Tofino Security, says one PLC vendor his firm works with estimates that less than 10 percent of its customers download the PLC patches it issues. "That's download, never mind install them," Byres says.

When Tofino issues security patches for its ICS security products, the company pushes the fixes as a free upgrade to its software in hopes of convincing more users to install the updates. "So if you install this patch with the update, it upgrades you to the latest and greatest," Byres says.

On two of these occasions, Tofino also directly contacted each customer in hopes of broader participation in patching. "We got less than 30 percent uptake," Byres says.

He's skeptical that SCADA patching will become the norm. He tells the story of a large oil company (which he would not name) that purposely runs a distribution control system that's three revisions behind the updated one. Why? The firm wants to be sure it has validated those versions of the software. "They'll use that patch in two years. There's the bad news," Byres says.

Timing is everything: The most realistic scenario today is these organizations patching semiannually or annually when they hold their regular maintenance shutdowns. "The best you can hope for is they do patching at an annual or semiannual shutdown," Byres says.

The likelihood that customers will apply patches to their SCADA systems is low, he says. "It's also completely a career-limiting move for an engineer if he installs a patch to fix something that isn't obviously broken, and it ends up shutting down the plant," Byres says.

SCADA patching was a rarity before the discovery of Stuxnet. The U.S. government ICS-Computer Emergency Response Team (CERT) had published only five security advisories in 10 years before the July 2010 revelation of the sophisticated malware attack on Iran's nuclear facility. In 2011, US-CERT posted 104 advisories for 215 publicly disclosed security flaws affecting some 39 vendors, according to a report Byres recently authored on the relevance of patching SCADA systems.

"Typically these vulnerabilities are publicly disclosed prior to the ICS vendors having patches available for the affected products. Furthermore 40% of disclosed vulnerabilities included working attack code. Individuals wanting to attack a control system can download exploit tools and run them against a target with little understanding of control systems or the consequences of their actions. And download and attack they do - ISC-CERT reported over 20,000 reports of unauthorized internet access to control systems in the last half of 2012," according to Byres' report.

Siemens Post-Stuxnet
Siemens, whose Simatic WinCC and PCS 7 systems were targeted by Stuxnet, has been a favorite target for security researchers looking for bugs. The SCADA vendor has been responding with some patches, as well as building security features such as firewalls and VPNs into some of its new products.

Alan Cone, product marketing manager for HMI software at Siemens, says SCADA customers are loathe to risk plant disruption. "It's a unique environment. When you have a plant up and running, you don't want to do anything to jeopardize the running of the plant. People are hesitant to make any changesm-- if it's not broken, they don't fix it," Cone says.

Siemens also has to test any Microsoft updates in its Windows-based products. "With our WinCC software for SCADA, [for example], we try to make sure we keep current with current releases from Microsoft," he says, testing Microsoft patches before adding it to its own products.

But Cone says Siemens customers overall aren't regularly practicing a patching process for their Siemens software. He doesn't have any data on patch downloads, however. "We're seeing customers paying attention to [patching] and evaluating it on a case-by-case basis," he says. "Depending on what patch it is, it could be major work to" apply it, he says.

When Siemens issues a patch, it also reaches out to customers directly. "We make sure we talk to customers about these things. If it's a really important [patch], we make a hard push. We have gone to some service packs and burnt CDs and DVDs" to push them the updates rather than relying on customers to download them, he says.

Some customers apply the patches, but others opt to hold off. Siemens also offers patch management services to its customers, he says.

If SCADA customers aren't patching their systems, then what's the point of vendors patching? "I wouldn't say patching is not the answer. It may not be the first answer," Siemens' Cone says. "Each [organization] has to look at the application and evaluate each patch on a case-by-case basis: 'Is this of concern to us? Is this something we expect to have, and how fast can we expect to put it in there?'"

Dale Peterson, CEO of Digital Bond, says major players in the ICS world patch on a quarterly basis, mainly on servers and workstations. "Our guidance to our clients is the first thing you should be patching are services that are exposed to external networks," Peterson says.

He's seeing more of these big organizations start adopting virtualization to assist in testing patches. One municipal water authority was patching monthly because it was using virtualization widely as a way to carefully test and roll out fixes.

But a system that's not secure really can't be upgraded to be secure, anyway, says Chris Wysopal, CTO at Veracode. Writing cleaner and more secure code is the ultimate goal, he says.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.