Triton/Trisis Attack Was More Widespread Than Publicly Known

[UPDATED 1/16/2019 7:40PM with updated new comments from Schneider Electric]

S4x19 -- Miami -- New details have emerged about the 2017 Triton/Trisis cyberattack on a Middle East plant's safety instrumentation system -- including a missed opportunity to quash it two months earlier than its ultimate discovery, according to an ICS security expert who assisted in the incident response.

New information also shows that the attackers infected six engineering systems, not just two as investigators had reported, said Julian Gutmanis, who was working out of a major oil and gas organization in Saudi Arabia at the time of the attacks, in a presentation here at S4. The publicly revealed attack on Aug. 7, 2017, was not the first incident suffered by the victim at the hands of the Triton/Trisis attackers, he said. In June 2017, an emergency plant-process shutdown system was knocked offline by the attackers. He also confirmed that the Middle East organization victim of the Triton/Trisis attack was a petrochemical plant in Saudi Arabia - a detail that initially had been speculated but not publicly verified.

The June incident wasn't accurately identified as a cyberattack by the petrochemical firm's vendor, Schneider Electric, he said, nor did Schneider ultimately take the proper remediation steps to clean up and remediate the attack. As a result, he said, the Triton/Trisis attackers remained unnoticed in the plant network; it wasn't until the August attack that it became obvious hackers were inside of it.

While early reports by Schneider and investigators into the attack said that a single Schneider Triconex Emergency Shut Down (ESD) system was targeted and knocked offline by the Triton/Trisis attack, it turns out there were actually six infected Triconex Emergency Shut Down (ESD) systems machines, all of which suffered a rare shutdown. These so-called safety instrumentation systems provide emergency shutdown for plant processes to prevent physical threats when a plant process reaches an unsafe level. These systems are not typically under the domain of security teams but, rather, engineering teams; Triton/Trisis was the first known incident to affect the OT engineering department.

"The June investigation was insufficient," with Schneider attributing the attack to a mechanical failure of the ESD system rather than a cyberattack, said Gutmanis, who will be joining Dragos. "They should have investigated what occurred in the plant," he said. "So the attackers got another two months [in the network] unimpeded. They ran executables multiple times between June and July before the August incident."

Red Flags
According to Gutmanis, the June Triconex system outage occurred on a Saturday evening, a time when most engineers aren't typically in the plant. The petrochemical firm called in Schneider to assist in troubleshooting the Triconex system failure; the vendor pulled logs and diagnostics from the machine, checked the machine's mechanics, and, after later studying the data in its own lab, addressed what it thought was a mechanical issue. "The ESD was determined to be fully functional again and the operations restored," Gutmanis said. "But there were a number of big red flags going on here."

He said the vendor recommended moving the system to a secure reference architecture. "That's good for a new plant but not advisable in the midst of an incident because it could expose admin credentials and other assets in the domain to the ... [still-] compromised system," Gutmanis said.

The now-infamous August shutdown of the Saudi Arabian firm's Triconex ESD system, leading to the discovery of the Triton/Trisis malware, actually involved six of the controllers, he said. These safety systems handled sensitive operations, including a burner management system. "The worst case was the potential release of toxic hydrogen sulfide gases," he said.

The attackers likely didn't mean to trigger shutdowns in the Triconex safety systems, but those were the only red flags until Gutmanis' team and others, including Schneider, dug deeper into the incident and found the sophisticated malware targeting those engineering systems. Gutmanis' team was called in for a rapid-response engagement.

There were other clues of a spreading attack uncovered, he said, including Remote Desktop Protocol (RDP) sessions to the plant's engineering workstations from within the IT network. "The plant on paper had a secure architecture. But we identified a poorly configured DMZ infrastructure that allowed the attackers to compromise the DMZ [between the IT and OT networks] and pivot to the control network," he said. "The DMZ firewall configuration was insecure," and the organization's perimeter VPN had been compromised and infiltrated.

"We saw a lot of traffic across the DMZ ... there were beacons we could see from the control network," he said. There was Mimikatz Windows-hacking traffic spotted, which had been flagged by the victim organization's antivirus software, for example.

"The entire organization was compromised," he said, and it became an all-hands-on-deck cleanup with various IT and OT vendors on-site doing cleanup and reboots of their systems.

Gutmanis said while his firm shared its findings with Schneider throughout the investigation, it was a "one-way" collaboration because Schneider didn't reciprocate. The first his firm heard about Schneider's findings was when the company was onstage at S4 in 2017 giving its presentation of the Triton/Trisis malware.

Schneider's public sharing of its attack findings was unprecedented for an ICS/SCADA vendor. The firm gave details of a vulnerability in its Triconex Tricon firmware that allowed the attackers to grab control of the emergency shutdown system at the plant, using a remote access Trojan as well in the attack. 

In an updated and more detailed response to Gutmanis' presentation provided on Jan. 16, Schneider issued a new statement on the incidents, including the June attack:

"On June 2017, the end user suspected an issue had occurred with their safety controllers and took one Triconex system offline, completely removing the Main Processors, and sent them to Schneider Electric's Triconex lab in Lake Forest Calif. At this stage, the end user wasn't aware of a cyber incident," Schneider said. "At the time, Schneider Electric was not on site to review the safety controllers. Once they were removed from power, the memory was cleared and there was no way to conclude that the failure was the result of a cyber incident.

"We were only able to analyze whether the controllers were working correctly within their safety function. After we completed our analysis, we determined there was no fault with the system components and we returned them to the end user," the company said.

The August Triconex attack was different, the company said. "In this scenario, the end user suspected a cybersecurity incident. They requested our support and one of our local engineers responded within four hours."

The victim plant then hired FireEye as its incident response head "and requested we communicate directly and only with FireEye," Schneider said. "In turn, FireEye directed us to investigate only certain aspects of the attack, namely the intent of the malware, and to report our findings directly to them. We were required to analyze a large quantity of data over an extended period. We continually provided updates to FireEye as warranted and necessary, following standard industry protocols.

"Schneider Electric has been, and continues to be, transparent about the incident. We remain willing to collaborate with end users, third parties and cybersecurity partners to drive industry-wide cybersecurity awareness and prevention," the company said.

'Lucky'

Gutmanis, meanwhile, pointed out that the petrochemical firm "got lucky" in the end, despite expensive and multiple outages for at least one full week per affected plant within the site. "The intent of the attacker was to manipulate the integrity of the ESD controllers," but no catastrophic physical disaster occurred, he noted.

The Triton/Trisis attackers had been inside the victim firm's network since 2014, said Rob Lee, CEO and founder of Dragos. The way attackers targeted the six sites could have caused a loss of human life as well, he says.

OT managers need to have a well-rehearsed incident response plan and be able to detect lateral movement in their networks, he said. "And recognize that your vendors' [approach to response] may be outdated."

In an interview with Dark Reading, Gutmanis said he believes Triton/Trisis could have been a "testing ground" for other such attacks in OT. He recommended that OT operators have monitoring in place, perform auditing, and be prepared for an incident to occur, such as knowing who to call and how to get them onsite rapidly.

Gutmanis' new details on the attacks came as a surprise to many security experts at S4.

"It's a big deal that the attackers realized they hadn't been caught and had the opportunity to continue what they were doing without consequences," said Michael Toecker, owner and engineer at Context Industrial Security. "There was an original missed opportunity to catch the attackers when they were on the DCS [distributed control system]."

Phil Neray, vice president of industrial cybersecurity at CyberX, says the key lesson from the Triton/Trisis attack is the organizational breakdown between the organization's IT and OT network operators. "There were no clear definitions of which team was responsible for ensuring that security controls had been properly implemented and were actually effective," he said.

Related Content: