CISA Takedown of Ivanti Systems Is a Wake-up Call

COMMENTARY

This story was updated on July 10.

In the wake of the attack on Ivanti's VPN software, which prompted decisive action from the Cybersecurity and Infrastructure Security Agency (CISA), what can we learn? This incident raises new questions about exploit techniques, organizational response to security breaches, and the skyrocketing cost of downtime.

First, let's break down what happened. From what's been disclosed, the vulnerabilities in Ivanti's system, particularly its VPN gateway, enabled threat actors to bypass authentication and gain unauthorized access. By sending maliciously crafted packets to the VPN gateway, attackers had a free pass to infiltrate the system without needing to steal credentials. Once inside, they could export user credentials — including domain administrator credentials.

Attackers also exploited a second vulnerability to inject malicious code into the Ivanti appliance, allowing them access to the VPN persistently (e.g., maintaining malicious control despite reboot or patch). Mandiant and Dark Reading reporting indicated Ivanti's initial stopgap mitigations were insufficient to prevent bypass: "Mandiant researchers flagged activity that uses a bypass for Ivanti's initial stopgap mitigation technique."

CISA warned that "the Ivanti ICT is not sufficient to detect compromise and that a cyber threat actor may be able to gain root-level persistence despite issuing factory resets." This suggests there was a time period during which attackers could and possibly did maintain persistence even following Ivanti's suggested mitigations (though Ivanti notes that there are no on-the-record incidents of this happening). In any event, it is clear that, as the incident was unfolding in real-time, there was a serious risk that attackers had achieved persistent access and uncertainty as to whether proposed mitigations were sufficient to defeat it. This is why CISA had to act quickly, and it did.

An attacker's persistent access to a VPN gateway is especially dangerous because the attacker can now move laterally within the VPN, using the gateway's trusted position to gain access to critical credentials and data. The bottom line: An attack compromising the VPN is bad, but here, the attack enabled the takeover of stored privileged administrative account credentials, which is much worse.

In response, CISA intervened to let organizations know they should assume the theft of critical credentials given the nature of the breach. The bigger concern was Ivanti's apparent failure to detect the compromise, leaving attackers free to operate within a trusted zone, bypassing zero-trust principles, and posing heightened risks to sensitive data.

Prompted by the severity of the vulnerabilities and potential for widespread exploitation, CISA took further action by taking two of Ivanti's systems offline. This is an unusual safeguard that was made after careful assessment of the damage and risk.

CISA correctly concluded that the risk of theft of privileged administrative credentials stored in trusted enclaves was much greater than the downside of complete shutdown. The calculus was that safeguarding the system's crown jewels, the most powerful credentials, required immediate action to minimize the blast radius of the breach, since they could not be sure they could operate the system securely.

As it turns out, Ivanti later clarified that patches could have been deployed discretely (i.e., per appliance rather than system-wide), which would have prevented the need for an entire system downtime. Ivanti has described "widespread confusion" that resulted from the lack of clarity on CISA's instructions for how to implement Ivanti's mitigations. These miscommunications highlight the importance of having clear open channels during a crisis. Mixed messages cause unnecessary chaos.

Measuring Hard and Soft Cost

Entire system-level downtime is costly. The IT resources required to securely and smoothly administer shutdown and recovery often are compounded by the losses incurred from complete outages of services, user downtime, and downstream effects (such as customers or dependent organizations that experience service outages). Not to mention the reputational and service level agreement considerations.

In Ivanti's case, we may never really know the exact cost. At the high end, assuming a VPN is mission critical for a portion of the workforce, downtime is a stop-work scenario for that user population and is therefore very expensive. Downstream customers, businesses, and users are also affected. This should be a warning to those of us addressing the aftermath of an attack in terms of weighing the risk "wake" that is likely to result in downtime costs.

CISA’s downtime to risk calculation was founded on assessing the "blast radius" of the attack. In this case, lateral movement from the VPN gateway was relatively easier because of the gateway's naturally trusted position, and the ability of the attacker to export stored credentials — including for privileged accounts. 

The blast radius of this breach was especially large because attackers were able to steal stored credentials and use them to move laterally. Minimizing blast radius of attacks is achieved by building systems using the principle of least privilege (e.g., zero trust). However, a service that stores credentials is inherently one of the — if not the — most trusted service in any given system. It is therefore not surprising that CISA made the call to shut it down, rather than risk further compromise.

So, what's the takeaway? The exploitation of vulnerabilities in Ivanti's software is a reminder of the threat facing organizations in the digital age. It underscores the need for robust cybersecurity measures and proactive infrastructure design and response strategies to mitigate risks and protect critical assets. Reducing the number of high value targets in IT infrastructure is an important step that minimizes the blast radius of attacks and can therefore reduce the need for broad shutdowns when attacks do happen. Privileged account credentials and stored keys are among the highest value targets, and IT leaders should accelerate adoption of strategies and technologies that minimize or eliminate such targets. As organizations navigate the aftermath of this incident, collaboration, clear communication, and continuous vigilance is essential in safeguarding against future threats.