A CISO's Guide to Paying Down Software Supply Chain Security Debt

There has always been a tradeoff in IT between shipping new features and functionality versus paying down technical debt, which includes things like reliability, performance, testing … and yes, security. 

In this "ship fast and break things" era, accumulating security debt is a decision that organizations voluntarily make. Every organization has security tasks stuffed in their Jira backlogs for "someday" — things like deploying security patches and running the newest, most stable versions of programming languages and frameworks. Doing the right thing takes time, and teams willfully postpone these tasks because they are prioritizing new features. A big part of the CISO's job is recognizing those moments when security debts must be paid.

One thing that made the Log4j exploit so alarming for CISOs was the realization that there was this huge accumulated debt that wasn't even on their radar. It exposed a hidden class of security gaps between open source projects and the ecosystems of creators, maintainers, package managers, and organizations who use them. 

Software supply chain security is a unique line item on the security debt balance sheet, but CISOs can put together a coherent plan for paying it down.

A New Class of Vulnerability

Most companies have gotten really good at locking down their network security. But there's a whole class of exploits that are possible because developer build systems and the software artifacts they leverage to write applications do not have a trust mechanism or secure chain of custody. 

Today, anyone with common sense knows not to pick up a random thumb drive and plug it into their computer due to the security risks. But for decades, developers have been downloading open source packages with no way to verify that they're safe. 

Bad actors are capitalizing on this attack vector because it is the new low-hanging fruit. They realize that they can gain access through these holes, and once inside, pivot to all the other systems that have dependencies on whatever insecure artifact they used to gain entry. 

Stop Digging by Locking Down Build Systems 

The fundamental starting point for CISOs, endorsed in materials like the developer guide "Securing the Software Supply Chain," is to start using open source frameworks like NIST's Secure Software Development Framework (SSDF) and OpenSSF's Supply Chain Levels for Software Artifacts (SLSA). These are basically prescriptive steps for locking down your supply chain. SLSA Level 1 is to use a build system. Level 2 is to export some logs and metadata (so you can later look things up and do incident response). Level 3 is to follow a series of best practices. Level 4 is to use a really secure build system. By following these first steps, CISOs can create a strong foundation for building a software supply chain that is secure by default. 

Things get more nuanced as CISOs think about policies over how developer teams acquire open source software in the first place. How do developers know what their company's policies are for what's considered "secure"? And how do they know that the open source they are acquiring (which constitutes the great majority of all software used by developers these days) is indeed untampered with?

By locking down build systems and creating a repeatable method to verify provenance of software artifacts before bringing them into the environment, CISOs can effectively stop digging a deeper hole for their organization in security debt. 

What About Paying Down Old Software Supply Chain Security Debt?

After you've stopped digging by locking down your base images and build environments, now you need to update your software and patch your vulnerabilities, including base image versions.

Updating software and patching CVEs is super tedious. It's boring, it's time consuming, it's a chore — it's work. It's the "eat your veggies" of cybersecurity. Paying down this debt requires a deep collaboration between CISOs and development teams. It is also an opportunity for both teams to agree on more secure, productive tooling and processes that can help make an organization's software supply chain secure by default. 

Just like some people don't like change, some software teams don't like updating their container base images. The base image is the first layer of container-based software applications. Updating a base image to a new version can sometimes break the software application, especially if there is inadequate test coverage. So, some software teams prefer the status quo, essentially loitering indefinitely on a working base image version that is likely accumulating CVEs daily.

To avoid this accumulation of vulnerabilities, software teams should as update images frequently with small changes and use "testing in production" practices like canary releases. Using container images that are hardened, minimal in size, and built with critical software supply chain security metadata, like software bills of materials (SBOMs), provenance, and signatures, can help alleviate the time-consuming pain of daily vulnerability management in base images. These techniques strike the right balance between staying secure and making sure production doesn't go down.

Start Paying as You Go

What's uniquely unpleasant about security debt is that when you just keep filing it away for "someday," it typically rears its head when you are most vulnerable and can least afford to pay it. The Log4j vulnerability struck right before the busy holiday e-commerce cycle and crippled many engineering and security teams well into the following year. No CISO wants to have hidden security surprises lurking.

Every CISO should be making a minimum investment into more secure build systems, software signing methods to establish the provenance of software before developers bring it into the environment, and hardened, minimal container base images that reduce the attack surface at the foundation of software and applications. 

Deeper into this massive software supply chain security debt payoff, CISOs face a conundrum of how much they are willing to have their developers pay as they go (by continually updating base images and software with vulnerabilities) versus postponing that debt and achieving an acceptable level of vulnerability.