When Software Updates Get Hacked

On July 2, an attack that used a series of yet-to-be-patched vulnerabilities in Kaseya's Virtual System Administrator (VSA) server software quickly compromised about 100 business clients of managed service provider (MSP) JustTech, encrypting systems in the victims' environments.

The havoc took about two hours to wreak but nearly two weeks to undo. Late last week, La Plata, Maryland-based JustTech — one of dozens of Kaseya-using MSPs hit by the attack — put the final touches on cleaning up the damage from the event and making whole the businesses and government offices impacted by the attack. While JustTech's prior focus on backup and restoration procedures helped its clients recover, the question the company is now pondering is how to stop the next major attack from using another supplier to affect its customers, says Joshua Justice, founder and president of the managed service provider.

"In the coming days, we will be asking these questions of our security vendors and having many discussions around improvements so steps can be taken to reduce the risk of this happening again," he says.

Attack Trend
The attack against Kaseya — attributed to the Russia-linked REvil ransomware-as-a-service (RaaS) group — is part of a trend of cybercriminals and espionage operators targeting the suppliers of administrative software used by companies to manage their environments. Prior to the attacks on Kaseya's VSA server, ransomware and nation-state groups had successfully compromised remote-management provider SolarWinds, Ukraine-based accounting software M.E.Doc, and system management software Piriform. In each case, malicious code hidden in a software update allowed the attack to spread to a large number of the vendor's customers.

Two decades ago, large companies may have performed their own regression testing on patches and not allowed software vendors access to assets in their networks. These days software updates are often immediately applied and companies often fail to put security controls in front of remotely managed services. The Kaseya attack on July 4, the Independence Day holiday in the United States, led the Department of Homeland Security's Cybersecurity and Infrastructure Security Agency (CISA) to recommend companies move to manual patching for the Kaseya VSA server, as a temporary protective measure.

While some security professionals have questioned whether allowing automatic updates for all software merely turns the perimeter into Swiss cheese, going back to a manual updating process is no longer an option, says Chris Wysopal, co-founder and chief technology officer of Veracode.

"I think without automatic updates, a lot of software would not get updated at all, so I am certainly pro-automatic updates," he says. "It is a security feature, and like all security features, it needs to be implemented right."

The speed at which the attack on Kaseya VSA servers happened prevented most companies from reacting to news of the attack, except to shut down the systems.

Depending on whether the Kaseya server was being used to manage clients' systems, businesses targeted by the attack may have had only a few clients impacted or hundreds. Without good mechanisms in place to limit unauthorized access or detect anomalous behavior, targeted companies had to get lucky to not spread the attack to their clients, says Mike Hamilton, founder and chief information security officer at Critical Insight, a managed security service provider.

"Automation had to be used to effect an attack that is this broad and which happened this quickly," he says. If they were not "using good third-party remote management policy and process," then the companies had to rely on the attack through the compromise update failing to work.

Limiting Exposure
Companies need to limit their exposure to compromises through the supply chain while still gaining the benefits from suppliers. During the Ransomware Task Force (RTF) effort to game out the worst-case scenarios, amplifying attacks through supply chains was identified as a critical weakness, Shank says.

"It isn’t clear how best to respond, as the world — and enterprise operations — becomes more and more connected and codependent every day," he says. "Each of these connections can be a pathway for massively good things but also opens the door to a shared-fate scenario, where a security incident at your supplier is likely to also become an incident on your network."

The REvil attackers disguised the code that they dropped on downstream customers systems as a patch, dubbed "Kaseya VSA Agent Hot-fix," but their initial compromise of on-premise Kaseya VSA servers had nothing to do with Kaseya's update mechanism. Instead, the attackers used an exploit chain of three vulnerabilities to gain access to on-premise VSA servers accessible from the Internet.

Blocking the next automated step — where the attackers used the Kaseya Agent, AgentMon.EXE, to execute a privileged series of commands — means limiting the access that third-party systems have to their infrastructure and networks. SolarWinds and Kaseya usually have administrator access to the systems they monitor and manage, but refactoring the software and strictly applying least privilege could allow companies to more closely monitor what those remote monitoring and management (RMM) systems are doing.

Blocking a single vector through which attackers can gain privileges to broadly attack infrastructure is critically important to prevent flash attacks. Network segmentation could also play a role in slowing down future threats. Most companies are still recovering, but those that are not through the worst of the attack are looking for better ways to prepare for the next attack.

"Critical discussions with [our] team members and clients will be how we reduce the downtime, how we decrease the economic impact, and how we minimize the negative mental health effects on clients and our team members," says JustTech's Justice.

In the end, each participant could have taken action to prevent the eventual attack. As a vendor, Kaseya should have implemented workarounds as soon as possible, especially when thousands — or more — organizations were known to be at risk. MSPs could have put more stringent controls in place to prevent the compromise of one privileged server from being used to amplify the attack. And the downstream companies should have good security software controls in place to block ransomware attacks, even if the company has no technical staff. 

In the case of the REvil attack on Kaseya, some security software scanners caught the ransomware component, according to Sophos, which saw a significant spike in detections of ransomware on July 2, the day the attack was initiated.

However, companies also need to talk about what works and not focus on what mistakes companies made that allowed the attack to succeed, says Veracode's Wysopal.

"We don't actually care that much about successful attacks. The ones that we really care about [are] the ones that were near misses," he says. "What did people do that helped save them? Knowing that would really help educate everyone ... it is crazy that we don't talk about this more."