PyPI Shuts Down Over the Weekend, Says Incident Was Overblown

Following a temporary suspension of all new users and package uploads, the Python Package Index (PyPI) repository is back up and running. Many noted that the culprit was the flooding of the site with a glut of malicious packages -- but a PyPI administrator noted that there was no unusual glut, simply fewer people than usual to address the usual glut.

PyPI is the official software repository for Python, serving over 700,000 users and over 450,000 projects, according to the site's homepage. Its popularity has attracted not just developers but hackers who like to upload malicious packages as a first step in supply chain breaches.

Beginning Saturday afternoon (UTC), PyPI temporarily suspended new user and project registrations. "The volume of malicious users and malicious projects being created on the index in the past week has outpaced our ability to respond to it in a timely fashion, especially with multiple PyPI administrators on leave," the site's admins wrote in an incident report.

The statement raised eyebrows across the security community, with many news sites reporting the site as falling victim to either an anomalous wave of malicious activity or even an outright cyberattack. And, the research firm Checkmarx in a blog characterized the situation as part of an uptick in “actors publishing overwhelming amounts of malicious packages in several open-source registries.”

But Ee Durbin, director of infrastructure for the Python Software Foundation, tells Dark Reading that the actual circumstances of the shutdown were much less dramatic than they were made out to be.

"This weekend was just a matter of human capacity," Durbin says. "Effectively, there was just one PyPI admin available to handle reports out of the usual three, and they (I) needed a weekend."

As of the evening of May 21 (UTC), PyPI was once again operating as usual, with its administrative team available in force.

Why We Worry About Open Source Software Repos

At least some portion of the hubbub around PyPI's 30-hour shutdown can be explained by growing fears around the state of open source security.

"We've seen the number of attacks skyrocket over the past two years," says Peter Morgan, co-founder and CSO of Phylum. In the first quarter of 2023, Phylum analyzed 2.8 million packages published to popular repos like PyPI, npm, and Nuget, 18,016 of which executed suspicious code upon installation, 6,099 referenced known malicious URLs, and 2,189 targeted specific organizations.

Malicious packages run so rampant today that some hackers hardly feel the need to hide them anymore.

"More and more attackers are realizing how easy this is to do. It doesn't require any skill. You can download scripts off of the Internet and use them to pollute the open source supply chain," Morgan explains. "Also, it's costless. You don't need to spend any money. You can do it for free with anonymous accounts."

With software today, Morgan continues, "there are so many dependencies. All an attacker has to do is get one foot in the dependency chain to get a hold in your computer. So the defender [has a] massive disadvantage here. The attacker only has to win once."

By contrast, organizations that utilize open source software — read: all organizations — have a far more difficult time defending against even such low-level attackers, prompting calls for better package inspection, the development of new tools to track dependencies, and software bills of materials (SBOMs).

Those in charge of maintaining the repos acknowledge these issues as much as anybody. "Regular caution should always be exercised when installing from a public index, whether in your projects or on the command line with 'pip install,'" Durbin says.

Repos Make Changes to Battle Malicious Packages

Historically, repositories have struggled to keep up with their far more numerous adversaries. To assuage concerns, though, Durbin tells of how "we have exciting developments that will allow for much more sustainable and potentially automated handling of malware reports coming soon."

The Python Software Foundation also recently added a security developer-in-residence role, meant to improve Python security at large. And just a couple of weeks ago, Durbin announced that PyPI will bring on a safety and security engineer, whose job will be to focus on PyPI's security in particular.

Supply chain security in years to come will turn on our ability to keep public repos clean and protect ourselves when they're not. "Everyone is very, very focused on finding things that have vulnerabilities," Durbin concludes, "but software vulnerabilities are not what attackers are using to break into computers today. They're creating malicious packages."