Lessons From The Ukraine Electric Grid Hack

New analysis and details about the devastating and unprecedented cyberattack that resulted in a power blackout in a region of the Ukraine last December illuminate glaring holes in security and operations that could have thwarted the attackers from shutting off the lights.

Security experts from SANS today in conjunction with the North American Reliability Corporation (NERC)'s E-ISAC published an in-depth postmortem analysis by SANS ICS experts of the attack, based on details revealed by ICS-CERT late last month as well as other public information. Aside from the glaring question of whether the attack indeed was sponsored by the Russian government, most of the nagging questions of how the attackers were able to black out a portion of Ukraine’s power grid have now been answered. The smoking gun has been confirmed: the attackers used stolen user credentials to remotely access and manipulate the industrial control systems and shut down power for some 225,000 Ukrainian power customers on Dec. 23 of last year.

“I think that the puzzle pieces are together now,” says Robert M. Lee, a SANS instructor and ICS/SCADA expert, as well as co-author of the report. “We’re missing the definitive attribution ... but the technical details” are mostly fleshed out, he says.

Ukraine officials have accused Russia, an obvious suspect given the military and geopolitical conflict between the two nations over Crimea. But the US reports steered clear of confirming that the attacks were the handiwork of a Russian state-sponsored initiative.

One thing US officials have confirmed is that the attackers staged a well-coordinated attack that relied on deep reconnaissance over a six-month period after they first embedded themselves into the network of three regional energy distribution companies. The attacks went live within 30 minutes of one another, and there also were three other organizations hit by the attacks that didn’t suffer any disruption to operations.

Like most targeted attacks, the Ukraine power grid attack began with a phishing email containing a malware-rigged attachment. In this case, Word Documents and Excel spreadsheets that when opened by users in the companies’ business network, dropped BlackEnergy3 malware that lurked around and stole legitimate user credentials. The attackers then used stolen VPN credentials to reach the industrial control systems network, and remote access tools to control the HMIs and pull the breakers.  

The attackers covered their tracks and bought themselves time, too, by installing their own custom firmware on serial-to-Ethernet devices at substations in order to knock them offline, and using KillDisk to wipe master boot records of the systems they hacked as well as to delete some logs. They waged a denial-of-service attack on the power companies’ telephone systems as well, thwarting their ability to communicate. In one case, KillDesk overwrote Windows-based HMIs in remote terminal units. The attackers also remotely disconnected Uninterruptable Power Supplies (UPS) systems to cripple power-restoration operations.

“It was extremely well-done -- how you would expect a well-funded team to operate,” Lee says.

In an interview with Dark Reading last month, Phyllis Schneck, the deputy under secretary for cybersecurity and communications with the Department of Homeland Security, said members of ICS-CERT’s team had been invited to Kiev to study and learn more about the attacks. “They spent four days working with our Ukraine counterparts to understand what happened,” she said. They learned that BlackEnergy malware was widespread in the victim networks, and the attackers “had their way with the systems” using stolen credentials, she said.

ICS-CERT’s findings showed how such an attack could “happen to anyone,” she said, and the agency wanted to provide recommendations for preventing such an attack on US critical infrastructure.

DHS undersecretary for the National Protection and Programs Directorate (NPPD) Suzanne Spaulding says she hopes the report will be a reality-check for US critical infrastructure owners. “I want ... [executives to say], ‘what are we doing about this?’” to prevent similar attacks, she said during an interview last month with Dark Reading.

There are plenty of lessons to be gleaned for power grid and other critical infrastructure operators in the US and around the globe.

For one thing, a cyberattack that results in a power outage takes some heavy lifting, and a bit of time, to pull off. “It took them six months or more to figure out these environments ... And it was only a partial outage,” says Lee, who notes that their methods weren’t necessarily sophisticated but were definitely coordinated. “We consistently see [the] theme for attackers who do the things we care about most in ICS networks ... it’s much more difficult” for them to do damage and it takes time, he says.

And that’s lesson number one: if attackers need a sufficient period of time for reconnaissance and learning the environment in order to control industrial equipment, the good news is that there’s actually a window for detecting their activity -- and stopping them from doing damage.

Network security monitoring could have helped spot the attackers before they shut off the power.

The Ukraine power grid attackers hid in plain sight for six months, gradually gathering enough intelligence and and knowledge to figure out how to access and manipulate the HMI and turn out the lights. Had the power companies been running network security monitoring tools, they could have spotted that activity.

There are many free and open-source network security monitoring (NSM) tools out there that can spot all kinds of bad activity in an ICS/SCADA environment, including unusual file traffic, a PLC code update, or command and control communications. Rob Caldwell and Chris Sistrunk ICS/SCADA experts from FireEye Mandiant recommend NSM for plants, and say NSM would have caught Stuxnet, for instance, and could be set to catch BlackEnergy. Some of the more popular tools come via the Security Onion Linux suite including Wireshark, NetworkMiner, Bro, and Snorby.

“If they had used network security monitoring practices, they could identify any reconnaissance ... and multiple VPN connections at times that were not normal,” SANS’ Lee notes.

Monitoring tools would have detected unusual data flows, something that’s relatively easy to spot in ICS networks because data flows are mostly static and predictable, he says. “When attackers are trying to learn [the environment], they disrupt those pathways.”

The attack punctuates the danger of remote access to ICS/SCADA networks.

VPN connections between the Ukraine power companies’ ICS and enterprise networks did not appear to use two-factor authentication, according to the report. “Additionally, the firewall allowed the adversary to remote admin out of the environment utilizing a remote access capability native to the systems,” the NERC SANS report says.

Ralph Langner, founder of the Langner Group, says critical infrastructure operators shouldn’t allow remote access to these systems.

“Limit remote access only to the people who need it,” SANS’ Lee says.

The report recommends using multi-factor authentication for any remote access communications.

Uninterruptible power supplies need protection, too.

The attackers commandeered a remote management interface to the UPS systems to schedule an outage for power at the energy company’s own buildings or datacenters.

“The online command interface to UPSes is another stupid flaw. These UPSes are located within the same building, so by controlling them via the network you just save five minutes for a maintenance job,” says Langner, who notes the CLI most likely would have been an embedded Web browser. He recommends disabling remote command interfaces to UPS systems.

The attackers also generated a DoS of thousands of phone calls to the energy company’s call center to derail restoration and communications.

“The reconfiguration of the UPS and the telephone DDoS: those two things added to the confusion, and to make the Ukrainians look incompetent. Those are two things I wouldn’t have predicted would have happened” in at attack like this, he says.

While the disabled UPS system and the telephone system DDoS were separate from the blackout portion of the attack, the goal appeared to be to embarrass the Ukrainians as well as to thwart restoration, he says. “During this attack, there seemed to be elements that highlight incompetence ... I think that’s interesting.”

Lee points out that the Russian media for the past year and a half has been reporting on the “incompetence” of the Ukraine infrastructure, and how they need Russia’s help. “The consistent theme [in the cyberattack] was not only being highly sophisticated in logistics and planning, but also in this showing” perceived weaknesses in Ukraine’s management of the power grid, he says.

Attackers can install malicious firmware on industrial equipment.

DHS in 2008 issued an alert to ICS/SCADA operators about a vulnerability in ICS/SCADA firmware update processes dubbed “Boreas.” It basically leaves an industrial systems’ firmware updates open to abuse, where an attacker installs his own malicious firmware to sabotage the system.

That’s basically what happened to the serial-to-Ethernet gateways in the Ukraine attacks, according to Langner, rendering them inoperable such that the operators were unable to communicate with the substations.

SANS’ Lee says the the custom firmware installed on the Ukraine networks’ serial-to-Ethernet gateways to “brick” them and disrupt the restoration of power was most surprising element of the attack. “That was extremely clever and it hurt the restoration effort of the Ukrainians,” he says. “I didn’t think we’d see an adversary clicking the breakers open and with what happened with the firmware.”

The gateways, or converters, basically translate communications between the serial protocols at physical substations and the overall Ethernet network that connects them. “By opening the breakers and modifying the firmware on those devices, it makes them unusable. In essence, they blew the bridges” up, Lee explains.

“They were cut off from the remote sites and had to physically drive out to them.”

Without a ‘cyber’ element to incident response and disaster recovery, a cyberattack is a disaster.

The Ukrainian power companies had no way to maintain control of their ICS/SCADA environment after the attack. That was an “eye-opener,” Lee says, and shows the crucial need for a “cyber” element in incident response and disaster recovery plans.

“You know they are opening breakers, so how do you quickly disable those features ... No one has that capability,” he says of ICS/SCADA operators.

That type of contingency planning is a big piece of the security picture, and until now, there’s been no experience in fighting back and regaining control when the bad guys have taken over, he says.

“There has never been a public case where the power grid was [affected] due to a cyberattack. This is the first time it’s happened, and it’s our only case study of what it looks like.”

Meanwhile, the lights may be back on in the Ukraine, but the nation remains vulnerable to another attack, Lee says. “It takes a long time to change processes, systems, and [get] trained personnel,” he says.

Related Content:

Find out more about security threats at Interop 2016, May 2-6, at the Mandalay Bay Convention Center, Las Vegas. Register today and receive an early bird discount of $200.