Power Utility Substations At Risk

Nearly 30 security vulnerabilities so far have been found in products using a popular ICS/SCADA communications protocol, prompting about half of the affected vendors to patch their products, and at least one vendor to pull its affected software from the market and urge its customers to instead install another one of its products.

The findings by researchers Adam Crain and Chris Sistrunk of potentially dangerous bugs in ICS/SCADA products running the so-called DNP3 protocol -- used for "master" host systems to communicate with equipment at power plant substations -- could be easily exploited by an attacker to disrupt parts of the power grid by crashing the master system so it can no longer monitor and control the SCADA network at a substation or substations. The attacks would entail sending malformed DNP3 response packets back to the master host system by exploiting flaws in the way software using DNP3 is written and deployed.

Cooper Power Systems, which was notified by the researchers of an improper input validation flaw in its Cybectec DNP3 Master OPC Server software, discontinued the server product rather than patch it, and is urging its customers to use its SMP Gateway product -- which doesn't carry the flaw -- as a replacement. The bug could allow an attacker to crash the system and, ultimately, disrupt the process it was running.

Last week at the S4x14 conference in Miami, Crain and Sistrunk disclosed new details on the so-called Project Robus research that they quietly began in April 2013. The researchers have been using Crain's homegrown fuzzing tool for DNP3 implementations, and so far have reported some 28 flaws, resulting in 16 security advisories from the ICS-CERT and related vendor patches. Only two products that the researchers tested have not had DNP3 flaws, and the researchers are awaiting word on nearly a dozen additional bugs that they have reported.

Some 75 percent of North American power facilities run DNP3, which was developed in 1993. The protocol is used for "master" servers to communicate with remote terminal units in electric substations, gas pumping plants for gas pipelines, and water utilities, for instance. That includes monitoring voltage or water levels, for instance.

Sistrunk, an engineer with an electric utility, a few months ago decided to try out Crain's open-source DNP3 fuzzer in his lab. "I tested it on a few things I have access to that had DNP3, and they broke. So I said, 'Time out -- we need to have a pow-wow and talk about what we're going to do because this is pretty big,'" says Sistrunk, who conducted the DNP3 research independently of his utility company, which he ask not be named.

An attacker could exploit these bugs and take down a remote site such that the utility would have no visibility or control over it anymore, says Dale Peterson, founder and CEO of Digital Bond, an ICS/SCADA consultancy that hosts the S4 Conference. "What it really means is that someone can go to an unmanned facility and take out the visibility of the entire SCADA system ... There's no need to go to the control center. They can pick [a power substation] in the middle of nowhere, go and break in, hook something up, and the whole thing goes down," Peterson says.

Sistrunk and Crain said that they also have found 90 or so DNP3 devices exposed on the public Internet. "The majority are misconfigured ... this is the [tip] of the iceberg. How many are on the Net that don't say anything?" said Crain, who is CEO of Automatak and the principal author of the Open DNP3 stack.

The exposed equipment is yet another example of the millions of public Internet-facing equipment found vulnerable and wide open to attack. Project SHINE, which has been gathering data on SCADA/ICS devices from SHODAN for a year-and-a-half, has identified more than 1 million unique IP addresses to date, and 2,000 to 8,000 new devices each day. According to Bob Radvanovsky, one of the Project SHINE researchers, the devices contain buffer overflows, misconfigurations, and cross-site scripting flaws, among other vulnerabilities.

[A global Internet-scanning project focused on finding SCADA/ICS equipment and systems accessible via the public Internet is discovering some 2,000 to 8,000 new exposed devices each day. See Project SHINE' Illuminates Sad State Of SCADA/ICS Security On The Net .]

The good news is that patching DNP3-based systems doesn't come with the baggage and risk of patching a PLC or other plant-floor system, where patching comes with risk of shutting down critical systems if a newly patched system goes awry. "It wouldn't be that much of a headache. I think that's an important point: We're not talking about the systems in the substations. We're talking about the master servers," says Ralph Langner, founder of Langner Communications, an ICS/SCADA consultancy. "It's like average IT equipment running a Microsoft OS."

And it's a relatively small number of "master" systems that are set up with redundant systems so that taking one down doesn't take down an entire plant, notes Digital Bond's Peterson. "I would expect to see something like this being patched. There's no excuse not to ... I expect over the next year or two a large percentage will apply the patches."

Next Page: Patch Missteps

But that doesn't mean the sites with Cooper's now-defunct and vulnerable system, for example, will swap it out for the vendor's more secure product right away. "It'll be out there until it fails. Owner-operators, especially with the recent economic climate, will run things until they break," Sistrunk says. "Jiggling the red wire is cheap. It's cheaper than replacing [a system]," he says.

One of the vendors whose code was found vulnerable was Triangle Microworks, a.k.a. TMW, which sells source code for deploying DNP3 to several SCADA vendors. The company has since patched its DNP3 Master and Outstation Source Code Library packages. The improper input validation bug could allow an attacker to remotely send a malicious TCP packet from the master station on an IP network, sending the substation device into an "infinite loop." An attacker with physical access to the master station could do the same thing to a serially connected device.

"It would be missing data and run until someone notices," Crain said. "If an operator isn't directly on that server, depending on how the software is architected internally, he might never know, and it might remain deadlocked until someone notices the data has not refreshed."

Not all of the vendor patches have gone smoothly. One vendor initially released details of the bug in its release notes, and two others issued so-called "silent fixes," meaning that they didn't coordinate their patches with ICS-CERT or notify their customers of the problem. And another vendor attempted to fix the flaw a couple of times, but ultimately "gave up," Sistrunk says.

There are some ways to mitigate DNP3-borne exploits, including not allowing DNP3 networks to touch the corporate firewall, ensuring strong physical security at substations, and ensuring that third-party software is tested for security before you buy it, Sistrunk says. "Ask for secure authentication and encryption," he says. "You can do SIEM on the enterprise side and just watch and know if you see something is wrong."

Meanwhile, the DNP Users Group board of directors emphasized in a document about the vulnerability reports that the problems were not with the DNP3 protocol itself, but rather with software implementations of it. DNP3 includes secure authentication in the application layer for both serial and IP communications.

The users group worked with Sistrunk and Crain on guidance for application developers on DNP3, which is available here (PDF) for download.

"The DNP3 protocol is sound. It is indeed possible to write a robust DNP3 implementation. This is evidenced by the few devices that Crain and Sistrunk were not able to crash with the Aegis fuzzer," says Jacob Brodsky, Chair of the DNP Users Group. "However, there is no denying that the DNP3 protocol is subtle and complex, mostly because SCADA systems themselves are a subtle and complex endeavor."

Sistrunk says deploying secure authentication doesn't preclude all of the bugs he and Crain discovered.

Have a comment on this story? Please click "Add Your Comment" below. If you'd like to contact Dark Reading's editors directly, send us a message.