Nation-State Attacks Force a New Paradigm: Patching as Incident Response

Patching security vulnerabilities has always been the most important security activity an IT team does. For the 25+ years I've spent in security, keeping systems up to date with security patches has been recommendation No. 1 in any set of IT best practices. And during most of this time, we have had the luxury of patching at our own pace.

We've agreed that 30 days to apply patches is the standard of good practice and that the IT team should expedite applying critical severity security patches. But now, after a three-month period when zero-day exploits were identified for SolarWinds, Accellion, Exchange, Chrome, iOS, Android, BIG-IP, and more, and with 11 zero-days identified in just one week, we must accept the reality that the old best practices are just not good enough anymore.

Every time we begin the next wave of incident responses (IRs) after each zero-day exploit is identified in the wild, we send out urgent messaging to help address these critical emerging threats. And the primary thing I've said over and over again is that any security patch for a zero-day or that addresses a critical severity vulnerability must be treated as a Level 1 security incident.

Because the bad actors know that most organizations do not patch faster than 30 days, and a huge number do not patch well at all, it's open season for the nation-states and their criminal advanced persistent threat (APT) groups to literally lay waste to wide swaths of industry and government. Our current approach to address these threats is failing in epic fashion right before our eyes.

Why It's Better to Risk an Outage
Security practitioners have long sympathized with the risks IT faces in emergency security patching. We have even tacitly suggested that, since it is a massive problem to risk bringing down critical systems and affecting operations when deploying a patch, it's OK to move methodically, take the time to test thoroughly, and plan a measured rollout. The bad actors' tactics are forcing us to change, no matter how strong the resistance. Once a zero-day is in the wild, every potential threat actor on the planet will sprint at top speed to use that zero-day exploit as widely as possible.

And if you wait to patch, you will be victimized and likely suffer a massively disruptive IR event that is likely to flatten the systems you were trying so hard to keep operating, with the data on those systems staged for sale on Dark Web auction sites. Ransom demands are hitting tens of millions of dollars — especially if more than one APT group is actively exploiting the same vulnerability.

Even if you pay the ransom, you'll spend $20,000 to $500,000 or more in response and rebuilding costs, not to mention the expense of service disruptions, lost revenue, fines, audits, higher insurance premiums, public relations campaigns to repair your reputation, and more.

In nearly every case, if the victimized organization had treated the zero-day vulnerability patch as an emergency, Level 1 security incident, activated their IR plan, and developed and implemented a real-time method to rapidly test and deploy the security patch, they would have avoided the costs of rebuilding from the ground up. They could also have avoided disruption of business operations, loss of service and revenue, lower customer retention, tarnished company reputation, increased regulatory scrutiny and potential fines, etc.

Incipient Security Events Require a Level 1 Response
It is vastly less expensive, less disruptive, and less impactful to call a cybersecurity incident and activate an IR plan that includes a playbook for responding to an incipient security event whose remediation is emergency testing and patching of systems.

The paradigm shift here is the concept of an incipient security event — one that hasn't happened yet, but it will, if given enough time. And that window before the bad actors try to use the exploit against your assets has already started to close.

IT and security operations must adopt the concept of an incipient security event that requires a Level 1 incident response. The inconvenience of an Exchange outage over a weekend due to a patch is insignificant in cost and impact when compared to a cyberattack on that same Exchange server that takes down every system on your network.

This is the new reality. This is not the security team harping about good cyber hygiene; this is a new paradigm forced by attackers who are more sophisticated, organized, and motivated and have found an attack methodology that pays off like a rigged slot machine.

Crucial severity vulnerabilities must be treated as incidents requiring an emergency response. This means changing foundational concepts in IT management and building something new. But because we cannot control the attackers' methods, and they have found the holes in our IT processes, they will bury us if we fail to respond.