Southern Company Builds SBOM for Electric Power Substation

S4x24 – Miami – Energy giant Southern Company in the past year set out to inventory all of the hardware, software, and firmware in equipment running in one of its Mississippi substations in an effort to create a software bill of materials (SBOM) for the operational technology (OT) site.

The SBOM experiment began with Southern Company's cybersecurity team traveling to the Mississippi Power substation to physically catalog the equipment there, taking photos and gathering data from network sensors. Then came the most daunting — and at times, frustrating — part: acquiring software supply chain details from the 17 vendors whose 38 devices the utility identified in the substation during its reconnaissance mission.

Alex Waitkus, principal cybersecurity architect at Southern Company and head of the SBOM project, said collecting and correlating information on all of the hardware, software, firmware, and interdependencies in the systems for the SBOM supplied the energy company with a deeper understanding of all of the software components in the substation and their potential exploitable vulnerabilities. Prior to the project, Southern had visibility into its OT network assets there via its Dragos platform, but software details were an enigma.

"We had no idea what the different versions of software we were running" before launching the SBOM project, he said in a presentation here this week at the S4x24 ICS/OT security conference. "We had multiple business partners who managed different parts of the substation."

That meant consulting with physical security and maintenance teams, for example, to assist in the utility's gaining the full picture of the substation's software supply chain. Southern cataloged a total of 18 control network system devices from two vendors; eight physical security devices from three vendors; four cybersecurity and telecommunications devices from four vendors; eight OT monitoring systems from six vendors; and one OT fault system device from one vendor.

The next step in the project was gathering SBOMs from each of its vendors represented in the substation. But there were roadblocks, as nearly 60% of the vendors Southern contacted for their products' SBOM information declined to provide the utility with the information. And on average, it took 60 days and a dozen meetings for Southern to actually receive in hand SBOMs from the cooperating vendors.

"So we were updating firmware [in some cases] by the time we received" the SBOM, Waitkus said. That meant an outdated SBOM, added Waitkus, who co-presented Southern's findings here at S4 with Matt Wyckhouse, CEO of software supply chain risk vendor Finite State. The project spun out of an OT SBOM roundtable organized by Wyckhouse at last year's S4 conference.

Wrangling the Software Supply Chain

SBOMs are a sort of list of the ingredients in a software program — the components and their versions — aimed at providing transparency in the software and its makeup to identify possible security weaknesses and vulnerabilities. Southern's project was especially challenging because creating an SBOM for an OT environment comes with more hurdles than for an IT network, mainly because industrial networks often have older software in legacy equipment critical to supporting uptime for industrial processes. And getting information on that old code is not easy.

"Supply chain transparency is an interesting issue that I never thought I'd get into," Waitkus told Dark Reading in an interview. But Southern's SBOM experiment, he said, illuminated three major security benefits for the power company: NERC CIP (North American Electric Reliability Corporation's Critical Infrastructure Protection) compliance management, vulnerability management, and software patching prioritization.

"When you have a lot of substation sites, a lot of people aren't always comfortable rolling out firmware updates remotely … so you have a lot of people going to sites to do work. With proper vulnerability management integration, you can prioritize patching through vulnerability and exploitability analysis — and potentially drive lower overhead" for those updates, he explained.

SBOMs also can provide visibility into a threat before it escalates, he noted. "Think Log4j: it took almost two years to get a full readout" on which software was vulnerable to the flaw, for example, he said. "SBOMs can help identify that and make it easier for them [software suppliers] to know if they have a Log4j in the future."

Waitkus said he envisions SBOMs as useful tools in the utility's procurement process as well, allowing Southern to gain deeper visibility into software products during the kicking-the-tires phase: "What's your third party code? What's your copyright license?"

In their presentation, Waitkus and Wyckhouse admitted the project didn't acquire all of the data they had hoped it would, and because of the lack of information from vendors, the overall quality of the resulting SBOM was not ideal.

Some vendors were eager to assist Southern in identifying their software, but most were resistant. For example, one vendor told Southern in an email that the SBOM information on its particular product was not available because it predates SBOM requirements under the time frame of the Biden administration's 2021 executive order for critical infrastructure security.

Trust but Verify

Southern didn't just take the vendor SBOMs they received at face value, however. Waitkus and his team created scripts to analyze the SBOMs for their accuracy to confirm they had correct component and code dependency data. In one case, Southern found in its firmware analysis that the vendor's SBOM was out-of-date, missing 72 components. Several other vendor SBOMs also were missing component and dependency data.

Software vulnerability information provided by Southern's vendors also required vetting to ensure it was accurate and timely, so Waitkus and his team mapped the SBOMs to vulnerability databases. Waitkus and his team ran vulnerability assessment and analysis tools to bolster the verification process and brought in independent vulnerability testers to weed through the bugs.

The goal was to ferret out vulns that were exploitable in the substation systems. In some cases, that meant that a vendor's own vulnerability exploitability analysis included with the SBOM didn't show the true picture of threats to Southern.

"We started to understand that the sensational number of vulns [in some vendor reports] really wasn't that big," Waitkus told attendees. In one open source package, for example, there were 3,000 vulns listed in the vendor-supplied SBOM, but Southern whittled that down to just seven that were actually exploitable, he said.

In another case, a vendor had pegged a vuln as nonexploitable, but Southern found otherwise in its own analysis. "There were public exploits on the interwebs for us to use" to prove it, he said in his presentation. "Trust but verify."

SBOM: A Work in Progress

Southern Company plans to operationalize the SBOM program but still has some work to do to before it can go to production. One challenge: It will require restructuring some vendor contracts to include SBOM requirements, Waitkus told Dark Reading.

"SBOM sharing is clunky right now," Finite State's Wyckhouse noted in the presentation. "If you're just collecting SBOMs and you can't do anything with the data, they are just JSON documents in a folder."

Meanwhile, a next phase of the OT SBOM project is about to kick off. Schneider Electric, MITRE, Ameren, EPRI, and Scythe, will join Southern and Finite State in a US Department of Energy-funded effort to automate some elements of the Southern SBOM project — including inventory, SBOM collection, verification, vulnerability and exploit analysis.