Cloud-Native Apps Make Software Supply Chain Security More Important Than Ever

Developers write code. That is how most people understand the job of a software developer. Reality, however, is more complex. While developers do write a lot of code, they almost never write all the code in the application. Where does the extra code come from? The Internet, of course.

Modern programming languages use building blocks in the form of packages to handle things like mathematics, text manipulation, and networking. This makes a lot of sense. There is no need for each programmer to write their own algorithms for basic operations. Many programming languages also support (and often encourage) modular programming, making plug-ins available to handle more complex, well-defined tasks. Over time, significant libraries of packages and modules emerged, written by the community, and shared freely on platforms like GitHub.

The primary reason that organizations leverage open source software is to speed up development. Building an application entirely from scratch is extremely rare, and so an estimated 99% of codebases contain open source components, and up to 70% of enterprise code is now based on open source. Developers are busy merging ready-made parts with custom code to achieve the desired result without reinventing the wheel. DevOps is similarly an assembly process of container base images, open source middleware, virtual machine templates, and cloud services such as storage, networking, and Kubernetes.

When all of that is accounted for, only a fraction of an organization's computing power will be running internally developed code. The rest will come from external sources. For security organizations, this introduces a universe of risks.

A June 2020 report on vulnerabilities in leading open source software reveals that their total number more than doubled between 2018 and 2019, from 421 common vulnerabilities and exposures (CVEs) to 968. Data collected by GitHub in October 2020 suggests that 17% of software bugs there were inserted intentionally by malicious actors. Together, these reports show that intentional pollution of open source projects, or "repo poisoning," is becoming a greater threat.

It takes an average of 54 days for a discovered vulnerability to be added to the National Vulnerability Database (NVD), making the infamous zero-day attack almost two months long. Even after vulnerabilities are reported, little time is spent fixing them. Developers surveyed by the Linux Foundation reported on average that only 2.27% of their time is spent on security bugs, leaving organizations with knowledge of the risk but no direct way to resolve it. 

Vulnerabilities, however, represent potential risk that can only be exploited when the vulnerable component is in use, unpatched, without compensating controls. To increase their chances of success, attackers have turned to poisoning open source projects with outright malware, intended to be incorporated into the software supply chain and executed alongside the application code. 

Popular package repositories like npm, PyPI, and RubyGems have become, for threat actors, a reliable and scalable malware distribution channel. During the past year, the number of supply chain attacks has surged by 430%. With DevOps, the reliance on external software is inviting more attacks targeting the supply chains specific to cloud native environments, like placing hidden malware in publicly available container images. Aqua Security's research team, Nautilus, recently revealed that several SaaS services used by container developers are susceptible to cryptocurrency mining. While the risks are real, organizations will not stop using open source software. The efficiency benefits are too great to simply go back and write software from scratch. Open source is here to stay.

Organizations intending on reducing these risks should start with gaining visibility and control over externally sourced software. Vulnerability scanners can help to manage the risks of security bugs in open source components. These tools must include automation, prioritization, actionable advice, and metrics to measure how vulnerabilities are remediated over time. Automation is important because relying on manual assessment and remediation processes takes too long and many developers lack the knowledge to prioritize the risks and address them.

Packages, ready-made modules, and even entire container images can also be the product of attackers trying to gain entry and sneak malicious code into applications. These tainted artifacts will not be in the NVD, will not be assigned a CVE, and will not be detected as published vulnerabilities. One example of a worst-case scenario is a misconfigured container image, with a default user of root, built on a base image containing a backdoor, and deployed in a cloud environment with open networking. 

To prevent such scenarios, organizations must consider their entire supply chain: how software enters the organization, from where, and by whom. There should be clear guidelines on the quality and reputation of open source software, preferring projects with recent commits, engaged contributors, and good governance.

Incoming software should not only be scanned for vulnerabilities but analyzed while running in a sandbox environment before being used. This will reveal malicious code that was inserted intentionally and will only be visible on execution. In addition, the inventory, version, and configuration of services in a cloud environment should be looked at as part of the supply chain, including the scripts used by DevOps to provision them.

Many information security practices were established at a time when development and operations were relatively trusted, but long-running servers were difficult to protect. Today, the state of security is reversed. Cloud native deployments, being small, immutable, and interchangeable, are easier to protect. But the supply chains that feed them require more attention than ever.