Disclosure, Panic, Patch: Can We Do Better?

Open source security has serious déjà vu — several times a year, a critical vulnerability in an open source library, component, or application places enterprises at risk, necessitating a quick assessment to find every instance of the component and a long battle to patch the issues.

In early December, for example, a member of the Alibaba Cloud Security Team found a critical vulnerability in the most recent version of a popular Java logging library, Log4j. Over the next month, companies scrambled to patch the issues as three more flaws were found in the open source component and attackers exploited the vulnerabilities to install ransomware. Next up, in late January, a vulnerability management firm found a serious — and trivially exploitable — local privilege escalation (LPE) vulnerability in polkit, the Linux component for setting security and access policies, allowing attackers to gain root on major Linux distributions.

While open source software projects need to keep the bar for new developers low, as projects becomes more popular and widespread, their maintainers need a greater focus on security and to find ways to eliminate vulnerabilities, says Eric Brewer, vice president of infrastructure and Google Fellow at Google Cloud. The current ad hoc approach is struggling to scale, he says.

"In the short term, what we are doing is companies — mostly, tech companies — put money in a pool and use it to pay people to find things and fix things," he says. "While we are plenty great at finding vulnerabilities, we are not great at fixing stuff. So the money that is needed is the money for fixing stuff."

A variety of industries and some public agencies are trying to do more. In January, technology companies and organizations met with the Biden administration and the heads of government agencies to discuss ways of producing more secure software and make patching less of a struggle. Google and Microsoft have both pledged billions to improve software security over the next five years.

On Feb. 1, the Open Source Security Foundation (OpenSSF) announced the Alpha-Omega Project, which aims to identify and aid the most critical software projects to create a secure development effort and close vulnerabilities. In the end — the "Omega" part — the OpenSSF hopes to shore up the security of 10,000 widely deployed projects. Open source software is widely used in development and widely deployed through information infrastructure, even when companies don't know its there.

The goal is to get companies to contribute to securing the infrastructure and paying for audits and to get developers to work on the mundane effort of securing software, says Brian Behlendorf, general manager of the Open Source Security Foundation (OpenSSF). While Microsoft and Google have both invested heavily into the effort, other companies will be needed to reach the 10,000-project mark, he says.

"It looks like 10,000 gives us a pretty complete coverage of the universe of open source software," he says. "But that is a finger in the air, and it will take more funding to ramp up to that. We are hoping to do enough of the right things to help folks, help people find enough sources of funding."

Hacking the Problem
The effort has a lot of work ahead. Even the first step — determining which software should be the focus of these early efforts — is a struggle. The OpenSSF's working group for securing critical projects has created a list of the top 100 open source efforts that should receive priority in funding and security efforts. The list includes major open source projects, such as the Apache Web server and Postgres database, and coding resources used by specific groups, such as Gatsby for React Web developers and NumPy for data scientists working in Python.

However, building a short list of critical software projects is not straightforward. In its second census of open source software and dependencies, a group of researchers at Harvard University and the Linux Foundation noticed that the list was dominated by JavaScript programs because that language's ecosystem relies heavily on importing external code. Based on its own data on the prevalence of vulnerabilities, researchers at Cyentia Institute, a data-science group, identified lesser-known packages — such as PyYAML and Lodash — that had the highest frequency of vulnerable installations.

Yet, while the Cyentia list had the Log4j vulnerability, none of the lists had all of the recent major vulnerabilities. The Log4j issue, for example, was added after the fact to the OpenSSF list of top 100 issues. Software components like the polkit security component that are included as one of the more than 2,000 software packages in a default Linux distribution, and may not have been considered as a critical component before its vulnerability.

It is a problem that gets really complex, really fast, says Sharon Brizinov, vulnerability research lead at Claroty, an industrial cybersecurity firm.

"You cannot take away the open source projects, because the dependency tree is so complex, and dependencies have their own dependencies, so there is no possible way to wrap your hands around all the dependencies and make sure its all secure," he says.

Bring Back Curation?
Part of the problem is the move to a vast system of distributed repositories that lack curation. In the past, an operating system vendor or an application maker would — ostensibly — test all of the components that go into the software, but as repositories became the norm, every project became its own island.

There is still room for curation, especially for long-term support distributions and commercial software, says Google's Brewer.

"If you are using software in a critical situation, you should be using curated software," he says. "You can curate it yourself, but most companies are not technically able to do that. That's why enterprises pay companies selling distributions — when you pay for support for Linux software, you are paying for a curated version of Linux."

Funding expert assessments of critical open source software, offering maintainers and developers better automation to enforce security, and working on way to speed patching are all ways that the security of the open source software ecosystem can be improved, says OpenSSF's Behlendorf.

Finally, companies should look at reducing the number of dependencies. A Google study of Log4j, for example, found that the majority of applications affected by Log4j included the library five levels down in the dependency tree.

"Part of reducing the disruption and cost of managing software is for companies to update their dependencies and minimizing the number they rely on for development," Behlendorf says. "Those dependencies are a form of technical debt that companies need to manage."