Why Patching Software Is Hard: Technical Challenges

First of a two-part post.

Like everyone else, I wish that Equifax had patched the software flaw that caused the breach right away. However, I also understand why this is difficult. I was an employee of one of the largest banks in the United States, with over 45,000 employees. For the first few years, I worked as a development lead for a team that created and updated applications that processed transactions for billions of dollars in investment assets. I later worked in public cloud, network, and security engineering roles helping teams across the entire company move applications to production.

Patching one software vulnerability on a few servers sounds easy. However, patching that one vulnerability in the context of thousands of devices, software applications, and software libraries across multiple locations and lines of business is another story.

Tracking Devices, Applications, and Software Libraries
Companies need to have an inventory of every single device that runs software. Equifax has close to 10,000 employees worldwide. Each person may have computers, phones, tablets, and other devices. The company must track every piece of hardware connected to its network and every virtual machine in a public cloud environment. Additionally, the company needs to know all software it runs, including operating systems, applications, and software libraries running on each of those devices. Some of the software doesn't have an automated update or notification process. Companies must vet the software to make sure that update process is not delivering malicious code, as happened in recent cases involving NotPetya, CCleaner, and malicious libraries in public Python repositories.

According to Crunchbase, Equifax acquired 16 companies between 1995 and 2017. Each time a company buys another company, myriad new technologies and software libraries are part of that acquisition. The acquiring company needs to make sure all software is up to date on the company systems it has acquired. Acquisitions involve many complex issues, and patching may not be a top priority. Merging different networks and IT systems is complicated and can take up to a year or longer. Acquisitions and restructuring may mean companies have different lines of business. Different people may manage software in various parts of the organization.

Updating Critical, Complex, and Legacy Applications
Many applications may share a single software library. Updating that library can break processes handling millions or billions of dollars in transactions. The company must test each application that uses the upgraded library before deploying a new version to production. In one case, it took a development team months to update a custom-built library to a new version of Java. The team had to test over 50 different financial processing applications that depended on that library before deploying them into production and removing the old version of the library.

Testing complex legacy applications can be challenging. Imagine all the rules related to US tax laws for a company that handles investment transactions. There are hundreds of variations that can occur that change the tax implications of a trade and what must appear on tax forms. The type of change made to the system will dictate how many of those variations a development team will need to test to ensure any tax or financial processing by the system works correctly. Hopefully, documentation exists for the application, or someone still works at the company who knows how to test infrequently updated legacy applications.

In some cases, installing and testing a patch is extremely risky. A software patch can break devices that cost millions of dollars, such as SCADA systems, medical devices, and research lab systems. No spare machine exists that system administrators can use to test the software update in advance. Patching the software may cause operations at an organization to cease. In the case of a medical facility, it could be a literal matter of life and death.

Patching Solutions and Alternatives
Just because patching is hard doesn't mean companies can ignore the problem. Organizations need to invest time and money into solutions that automate software deployments and track software inventory. If companies are not aware what software exists in the organization, they won't be able to make sure it is all up to date. When patching is very risky, companies can limit network access to the port that exposes the vulnerability or turn off the vulnerable features of the software.

In addition, companies should move legacy software to new software architectures with security designed in from the start. Companies can measure the return on this investment based on the cost other companies are facing due to massive data breaches. Additionally, if this keeps happening, companies should consider the cost of increased legislation designed to prevent data breaches — some of which may add overhead without solving the problem, like regulations related to PCI-DSS compliance. 

In the second part of this two-part post, I examine the organizational challenges involved. 

Related Content:

Join Dark Reading LIVE for two days of practical cyber defense discussions. Learn from the industry’s most knowledgeable IT security experts. Check out the INsecurity agenda here.