What the CEO Saw: Colonial Pipeline, Accellion Execs Share Cyberattack War Stories

MANDIANT CYBER DEFENSE SUMMIT - Washington, D.C. - Joe Blount, president and CEO of Colonial Pipeline, says as soon as he learned that his company had been hit by a major cyberattack, his day job took a back seat to the subsequent all-hands-on-deck incident response.

"Your typical CEO job went out the door just a few hours ago, and it's not coming back for quite some time," he said, describing what it was like when he was first informed of the ransomware attack, which led to the company temporarily shutting down its physical pipeline as well as OT and IT systems as a precaution, and ultimately paying the $4.4 million ransom. Much of that ransom was recovered later by the FBI - about $2.3 million of what the company paid to the DarkSide ransomware gang.

Blount, like most of his executive team and employees, was assigned a specific role in the company's response: He was the "conduit" for communicating with the US Department of Energy (DoE) about the attack details, response, and recovery. 

"In our case after the attack, the CEO responsibility immediately becomes to contain the attack and remediate the situation. That becomes the focus," he said.

Blount, along with Accellion chairman and CEO Jonathon Yaron, shared the CEO's view of a major incident response to a cyberattack here during a keynote panel with Mandiant senior vice president and CTO Charles Carmakal.

"After an incident like this, there is not enough time in the day or enough people. So you become actively involved yourself," he said. For Blount, that meant conducting daily update briefings with the federal government via DoE about what was happening and what Colonial Pipeline and its incident response team, including Mandiant, had found.

"When we set up that one conduit with the government - which allowed us to communicate all the way up to the White House, to every regulator responsible [for the industry], to all the way through to the lobbyist groups who were helpful in disseminating information to like companies," he said, it allowed them to indirectly alert other organizations of the threat.

Accellion's Yaron, a former member of the renowned Israeli Unit 8200 intelligence team, recalled the second round of attacks exploiting zero-days in the company's legacy File Transfer Appliance platform nearly a month after the first attack on the platform. 

"Here it is, two ex-8200 guys," he said, referring to himself and his company's head of technology. "We obviously understand somebody has outsmarted [us] in the second 0-day [attack] in late January," he said, and the attackers "know something we don't know."

The attack first was spotted when an anomaly detector in the Accellion FTA - a 20-year-old technology that was still used by some companies to transfer large files - fired an alarm at an academic institution in the northeast US, who then contacted Accellion. It was unclear to the vendor whether it was a government or commercial attack, and whether it was a single event or a mass event, he said. Banks, US government agencies, and a major healthcare organization were among the customers still running the older product.

"The first order was to understand the magnitude," Yaron said. There were some 300 possible victim organizations, but in the end, Accellion found that close to 90 were hit, 35 of which suffered "significant impact."

The breach at Accellion resulted in stolen customer data and, later, extortion attempts used as leverage by the cybercriminals. The vendor issued a patch for the first zero-day attack in December, within 72 hours of the discovery, and also urged customers to move to its current Kiteworks firewall platform. But on February 1, they revealed the attackers had been at it again using a second set of vulnerabilities in the platform.

Mandiant found data from companies in the US, Canada, the Netherlands, and Singapore had been dropped onto a Dark Web site with ties to the Russian cybercrime gang known as Fin11. Kroger, Jones Day, and Singtel were among the victims of the Accellion breach.

Accellion doubled down on urging customers to shut down the FTA systems. 

"The vast majority listened to us and shut the systems down," Yaron said. "That's why no more than 10% [of Accellion customers] got heavily penetrated."

'This Is Crazy'
One Fortune 100 customer declined to shut down its FTA system. They maintained their operations were too critical to interrupt. 

"'We're going to monitor it, second by second,'" Yaron recalled their senior management team telling him. "I said, 'This is crazy' ... [but] they succeeded in keeping the perpetrators out."

Colonial Pipeline's Blount says he was getting ready for work early on May 7 when he was told about the attack on his company. 

"I received word that we had received a ransomware attack through one of our systems in our control room," he recalled. "By the time that I was notified, we'd already gone about the task of shutting down 5,500 miles of pipeline. The employees are trained to do so when they perceive a risk; as you can imagine, we didn't know what we had at that point in time. We knew we had a threat, we knew that threat had to be contained, and therefore we shut the pipeline down in order to do that."

The shutdown was the standard response procedure when identifying a risk and remediating it. At that time early in the investigation, Blount said, there was no confirmation whether the IT or OT systems were at risk or whether the pipeline was at physical risk, so they opted to shut it down as a precaution. 

"We knew we had a ransomware attack, but did we potentially have a physical attack? Could it potentially be a nation-state trying to cause damage to the US? So we ramped up and had the pipeline shut down within an hour," he said.

Unlike most ransomware victims who pay up, Colonial Pipeline ended up getting most of its money back. The FBI's recovery of the ransom was "a huge win for us as a security community," Mandiant's Carmakal said.

Colonial Pipeline handed to the FBI its bitcoin wallet within a day of the payout, which helped the agency successfully retrieve the money, according to Blount. 

"The government was highly focused on helping us bring our systems back and to help alleviate a criminal attack on, frankly, the whole country," he said.