'It Saved Our Community': 16 Realistic Ransomware Defenses for Cities

Figure 1:

There's only one road in and out of Valdez, Alaska. The nearest city is Anchorage, a 300-mile drive away, and not a direct one. So the 3,976 citizens of Valdez -- that's "Val-deez" with a long "e" -- are used to handling emergencies on their own. According to police chief Bart Hinkle, the city was just about founded on disaster, famously hit by the Good Friday earthquake of '64 and the Exxon Valdez oil spill. But Valdez had never experienced a disaster quite like this one. 

July 26, 2018, while the town's new IT director Matt Osburn was out of town and his second-in-command was running a routine update, the Hermes 2.1 ransomware took hold of Valdez in the dead of night.

All records (including police), city administration, finance, planning and zoning, the port and harbor authorities, and "basically the entire city network" was locked down, Hinkle says.

And even though this small town was more prepared than most of the hundreds of others that have been felled by ransomware recently, and even though municipal leaders ultimately decided to pay the ransomware operators, there was no quick fix. The recovery process rolled into weeks, then months.

"Government is a different animal certainly than private industry," says Reg Harnish, executive vice president of security services for the Center for Internet Security. "Because they're spending taxpayer dollars, [municipal governments] have a lot of bosses."

Not only is funding generally short, cybersecurity has stiff competition for those funds with other essential services.

"You have to have conversations like, 'We can have cybersecurity or we can fix a bridge,'" Harnish says. How does a municipality translate the value of a firewall into the value of some addition to K-12 education, he says. "Politics plays a role ... introduces a lot of different pressure," Harnish says.

Ransomware can impact the availability of any one of these essential services, from court systems to payroll to water.

Omri Admon, cybersecurity expert from SOSA (the firm selected to create the Global Cyber Center in New York City), points out how ransomware can add insult to injury for cities that are already strapped for cash.

"It's an additional loss of funding if they can't process property taxes" and other sources of income, he says. "It's another layer of complexity that puts them in a chokehold.

So here's some realistic advice on how to avoid ransomware infections and what to do when one happens, courtesy of Valdez leadership and others who have witnessed municipal government ransomware infections up close. {Continued on Next Page}

(Image Source: stnazkul via Adobe Stock)

1. Buy Cyber Insurance
More organizations in both the public and private sector are investing in cyber insurance as part of their overall security strategies. That includes the city of Valdez.

"Whomever was responsible for getting the insurance for the city deserves recognition and/or a raise," says Valdez's Hinkle. "I can't emphasize enough how much it saved our community."

The mysterious savior who added cyber insurance to a broader policy helped Valdez pay for a wide variety of costs, including the ransom and the town's new IT infrastructure.

The insurer also had preferred provider relationships established with cyber incident response companies, forensic investigators, and other resources that could have been essential in an emergency.

(Simply having the policy, however, isn't enough, however. Calling the insurance company right away and documenting everything thoroughly is essential to getting a claim paid.)

2. Get a Third-Party Ransomware Risk Assessment Now
Earlier this year, the city of Baltimore opted for roughly $18 million of recovery costs over paying a ransom. In July, at the US Conference of Mayors, more than 14,000 US mayors signed a resolution agreeing they would not pay ransoms. A strong commitment to not pay would, hopefully, dissuade financially motivated attackers from going after governments in the first place.   

But let's not be too hasty.

"You've been violated, and you don't want to be a victim," says John Pironti, president of cybersecurity consultancy IP Architects. "But you have to take emotion out of the equation  because you have to make hard decisions based on cost."

A thorough risk assessment that outlines these costs — determining just what kind of outage is worth just this much of ransom, etc. — could help guide those hard decisions. And having that risk assessment done by a third party makes it easier to defend those decisions later against political opponents or public pressure. 

In Valdez's case, "I knew that I had all of my criminal cases encrypted," says Hinkle. "I had a homicide case -- all the evidence was gone. You name it. That's just one department. I knew that four Bitcoins, in my mind, was worth it and a good use of taxpayer money."

3. Identify Your Response Team Now
Check what official chain of command has already been established. Is the city comptroller, the police department, the mayor, or someone else the first call? 

When does the FBI get involved? What are your legal notification requirements, and when should attorneys and public affairs be called? Who hires the third-party advisers, the forensic investigators, and any other cybersecurity specialists?

Wouldn't you like to know these answers before your computer screen is locked and your phones don't work? 

Pironti, Admon, Hinkle, and Osburn all recommend identifying the people who should be part of the incident response team, FBI included, and building relationships with them now. 

"The longer we wait, the less evidence we have," Admon says. "So coming to authorities is extremely valuable." Quick reporting could help prevent infections at other sites as well.

4. Join Forces with Other Cities
By "join forces," we don't mean celebrate a hootinanny of a coordinated ransomware infection, as over 20 Texas towns did one day this month. We mean learn from one another and even share staff.

Valdez, Alaska, was aided by the fact that a nearby locale was hit by ransomware just a few days earlier and shared information about the attack.

Government agencies across the country complain about being outpriced by the high-paying private sector when competing for cybersecurity staff; the problem is even worse for small towns that not only have limited funds, but enough work to merit a full-time security specialist.

The Center for Internet Security's Harnish suggests small towns might address these staffing issues by "security through association." Towns like Valdez "could probably do better if they banded together with other surrounding villages," he says.

The Center for Internet Security and the Multi-State Information Sharing and Analysis Center (MS-ISAC), which serves the US's state, local, tribal, and territorial governments, provides information sharing, as well as free and low-cost educational resources.  

5. Awareness Training for Users & Taxpayers
Most ransomware attacks happen through human error, according to SOSA's Admon. "Just by educating, they can reduce the risk, but they're usually pretty bad at it," he says.

Admon suggests creating a simple security checklist for end users. Distributing and displaying a checklist of fundamentals will create a very basic layer of security that reduces the threat of phishing and human error that have caused many a ransomware infection.

Harnish suggests something even simpler. "One thing a small city can do is remind users, 'Hey, we're under attack [every day],'" he says. A little reminder in the morning and a simple request to be please be careful today could make all the difference.

"We really need to focus on human behavior as a solution, not technology," he adds. "Tech is a great backup." 

Spreading the word to the general public also helps to gain support for further investments. {Continued on Next Page}

6. Map Out Your Attack Surface
Municipal governments are generally so distributed, with a variety of diferent protections for different functions, and any one of them could be the source of a ransomware infection that spreads through a municipality.

Admon suggests city IT directors first map out all of those potential entry points. Look at what you have before you try to secure it.

As part of that effort, determine which assets are the most valuable. "When it's doomsday," says Admon, "this is the most important data." Then apply protections accordingly.

7. More Segmentation, Less Integration
Part of the worry with ransomware is with it spreading, says IP Architects' Pironti, so, "create logical chokepoints." If one area gets challenged by an attack, an insidious ransomware variant like WannaCry can't laterally shimmy its way across the entire network.

Separate operational systems from administrative systems, he advises. 

A few things the city of Valdez did not have to worry about: inoperable 911 services, SCADA systems, or phones. All of those systems were isolated from the rest of the IT networks that were impacted by the ransomware. 

8. Practice Good Security Hygiene
It's worth repeating: "Patch and config, patch and config, patch and config," Pironti reminds. "If you're not bothering to do patching and configuration well, then don't bother investing in endpoint control."

The basic, boring blocking-and-tackling of security will often be what saves an organization from any infection, he says.

Limiting administrator privileges and managing credentials properly will also help prevent the free-wheeling lateral movement that ransomware operators exploit. 

9. Do Better Backups, with Versioning
Remind employees to back up their files, Admon says. "It would make everyone's life so much easier," he says.

Pironti more specifically recommends versioniong backup solutions. They'll enable you to return to versions of files that existed before the ransomware attack, which makes recovery easier.

10. Then, Deploy Endpoint Solutions
Products provided by companies including CrowdStrike, Cylance, and Carbon Black might not be affordable for all of your city's endpoints, but they could be worth the investment for those that are most sensitive. {Continued on Next Page}

11. Get an Incident Command System Going
In the sad event that your city is overtaken by ransomware, an incident command system (ICS) will help you organize — providing your response team an action plan and a place to work.

The ICS will also give idle government staff new direction. According to Hinkle, when all the computers went down in Valdez, a lot of people who'd never lived in an analog world were rather at loose ends.

"There's a certain generation of people that don't know how to write. So without their computers, there was a lot of idle time and sitting on hands and 'what do we do now.' ... There was a whole workforce out there that was lost without a computer." he says.

"The ICS model really helped give those people tasks that they could do," says Osburn.

"More than anything it helped keep frivolous requests off Matt's desk," says Hinkle. 

12. Isolate Systems and Contain the Impact 
Because Valdez didn't know the entry point of infection, they also collected all employees' scattered thumb drives. In an abundance of caution — perhaps the attack was a distraction from a related data exfiltration attack — they froze all the city's credit cards.

"First thing I knew ... You do not want to power down systems," says Valdez IT director Matthew Osburn. "You want to isolate them. And that really saved us in the long run."

For example, even though financial systems were locked down by ransomware, Osburn's team was still able to access the SQL Server, which expedited their ability to get back to issuing payroll and tax receipts.

13. Check for Decryption Keys Online
NoMoreRansom.org maintains a list of published decryption keys. The one you need might very well be there already. Nevertheless, be careful. This might be better done by your professional cyber incident response team.

14. Watch What You Say to Ransomware Operators
Osburn made it very clear that nobody in the city of Valdez communicate with the attackers directly, just as other experts advise that victims let the experts lead communications with criminals. 

The city wanted to ensure that the decryption key would work before sending payment. But, the attackers, who often automate their attacks, did not necessarily know what victim they were talking to. So, when communicating about decryption keys, the Valdez response team protected the city's identity.

"[The attackers] said give us a couple filenames, we'll decrypt them for you to show we have that capability and that it works," says Hinkle. "And so we put our group to work finding three files of which I knew the content and which could not be traced back to the City of Valdez.

"Obviously you don't want it on letterhead. Anything that could be traced back. Once they learn that it's the city of Valdez they have, a simple internet search would reveal that we're able to afford more than 4 Bitcoin. Because up to this point they don't know if we're just a small little mom-and-pop grocery stand. ... They deal in bulk." 

IP Architects' Pironti also encourages negotiation.

"You never pay full price," he says. Many of the systems are automated, he says, so if you drop one Bitcoin in, even if they asked for five, they might automatically issue the decryption key. "In some cases, they're happy to get anything," he says.

15. Be Careful With Decrypted Files 

Osburn didn't just welcome the files back with open arms, though.

"We got our files back," says Osburn, "but I don't trust 'em."

He's carefully reintroducing files, making sure not to invite weaponized documents or new malware into the system. 

16. Learn From the Experience
Since the initial attack, Valdez has begun incident response plans and a wholesale overhaul of its IT infrastructure so it never happens again.

"Communities prepare for those natural disasters," says Hinkle. A ransomware outbreak "is one that should be tabletopped or practiced or put in place. There was not one for us. We did a lot of things very well, but this was literally learning as we went."

They have since sped up all their other planned upgrades. Osburn says that what would have been rolled out over three years has now been rolled out over three months instead. 

They've upgraded outdated systems to Windows 10 and Server 16. They've virtualized legacy bare metal servers. They're boosting their firewall and endpoint protections.

"We have a different backup system that's not integrated with Active Directory just for this reason," says Osburn. ""There were certain security procedures I was wanting to implement that were a little unpopular. One of them was more difficult passwords. Well, after not having your documents for a month, having your documents and a more difficult password seems like an easy compromise. [The attack] really paved the way to get us where we need to be." 

It's such a beautiful and completely necessary overhaul that Osburn seems secretly grateful to the ransomware operators.

"Usually these systems are very hard to upgrade because you can't have downtime ... Well, that decision was made for us."

"While I'd never want to go through this again, I'd say it's probably one of the best things that could have happened to the city. It gave us the opportunity."

(OK, maybe not-so-secretly grateful.) 

Related Content: