Hacking The Router Patching Conundrum

The dirty little secret about patching routers is that many enterprises don't bother for fear of the fallout any changes to their Cisco router software could have on the rest of the infrastructure. But the recent discovery of a way to easily hack the devices has turned upside down conventional wisdom that patching routers is more of a risk than an actual attack on these devices.

Researcher Felix "FX" Lindner's research earlier this year demonstrated that multiple versions of routers can be attacked -- specifically, Cisco's PowerPC routers -- shooting down the assumption that hacking routers requires separate exploits for each type of router. Enterprises traditionally have been content to avoid patching their Cisco routers because the chances of a major breach was less likely than the possibility of an unintentional outage from a router update.

"The underlying problem is that you cannot patch IOS -- you always need to update the entire image. And with this comes all kinds of compatibility issues with your configuration, hardware, and setup," says Lindner, a researcher with Recurity Labs.

Lindner demonstrated with his research that all an attacker needs is basic knowledge about the targeted device, rather than specifics of the IOS configuration. His exploit method applies to stack-buffer overflows, and he was able to execute memory writes and to disable CPU caches on Cisco routers running on the PowerPC CPU.

Router updates aren't typically a top priority, and few organizations have policies and procedures in place for patching their routers. "They're not thinking about all of the routers out there," says Dan Kaminsky, director of penetration testing for IOActive. "They're resource-constrained and overloaded: I get that. They need a good reason if they are going to deploy their limited resources to monitor yet another problem. And [Lindner] has provided a damn good reason.

"The idea that the variability of router platforms would defend you from an attacker is false. All versions have something in common [in this research], and this is not just in theory, but FX demonstrated it and used it to exploit all [PowerPC IOS] versions."

Even so, Lindner's groundbreaking research has yet to change the status quo. "For all enterprises and carriers that I know of and spoke to, nobody updates IOS when a new security vulnerability is found. The risk associated with upgrading IOS is, in fact, higher than the risk of getting 'pwned,'" Lindner says. "Most sensible network operations groups will try to filter the new issue on the border, if they still have something like a border, [and] the most advanced groups will have core-dump writing configured on their routers to catch exploitation attempts."

Cisco Systems says some of its customers patch, while others do not. "Our customers are all over the place -- some do patch diligently, and some have very strict policies," says Russ Smoak, director of technical services for Cisco. "We have the other extreme: those that have aged infrastructures as well."

Smoak says the threats to routers, in general, haven't changed much, though the reasons behind them have. Distributed denial-of-service (DDoS) attacks are typically more economically motivated now than they were before, he says. "Attacks are more subtle and more targeted," he says. "It's the same stuff, but different motivations behind it."

Cisco views its product as a target, Smoak adds. "We try to take a very paranoid view, and we do a lot of things to harden our products," he says.

Security experts agree that patching routers isn't easy, but some steps can help prevent taking down the network in the process. Recurity's Lindner says patching is, indeed, likely to cause something to break. "Many of your configurations don't work anymore, your line cards are not supported with the new release, or something else breaks," he says.

But if you keep your IOS minor version up to date, you can use the patched-IOS image, he says. Cisco could also provide some additional patch information: "What Cisco could do is test transitions from one image to the other on many platforms," Lindner says. It could publish "safe-to-replace lists," which would note that if you replace version 12.2(13) with 12.3(14), for example, certain features are not affected, he says.

For large organizations, patching Cisco routers should really be an extension of their redundancy policies, says Fred Avolio, senior professional staff with The Johns Hopkins University Applied Physics Laboratory. "You probably have redundant [routers] in place already. If you don't, you're going to have to. And you should do half of [the patches] at first and see if [they] take, and then do the rest later," Avolio says.

Assessing your risk is also important, Avolio says. "If this [vulnerability] only works on a particular configuration or version of IOS, that's part of your risk equation. But because it's more likely now that malware will be written for routers on various versions of [IOS]...the security policy you have in place needs to be adjusted because of this change in threat."

Not surprisingly, Cisco is one of those organizations that patches its own routers regularly. So how does Cisco handle the delicate router-patching process? Craig Huegen, director of IT network and data center services architecture for Cisco, says upgrading the network infrastructure involves several steps.

"First, there is a review stage, where an assessment is made of the software update. What are the known caveats, if any, to the new software? Are there any known bugs that the administrator should watch for? Have any system resource requirements changed, such as the memory or system image storage? Have any features changed?" he says.

Then the new software is downloaded and the image validated to ensure it's complete and unmodified. "In many cases, to ensure quality, images are loaded [and] activated on lab devices to ensure functionality and certify them for use," Huegen says. "Third, the new software is staged onto the production devices and prepared for use. Finally, during a scheduled maintenance window, the new software is activated [in a rolling fashion]."

Cisco gives critical security updates high priority because they could immediately affect the security or operation of the network, he says, and regular, noncritical ones are done within a regular deployment schedule. Any new feature updates to the routers are usually project-driven, he says.

"It would be a rare case that I would suggest you not need to update. For a very specific, very static application with very minimal access, it may be acceptable once long-term stability is proven through burn-in time," Huegen says.

IOActive's Kaminsky recommends that enterprises run Cisco's router-monitoring software, ensuring that branch offices and new acquisition or merger offices are also monitored. "And worry about hardware that's too old to be patched. You might have to buy new hardware," he says.

More food for thought: Consider automatic patching on the infrastructure. "We know that if we want widespread deployment of a patch, we need as little pain as possible," Kaminsky says. "You should front-load the pain into the testing process -- that's where the work is. That's the hardest part."

While there's no perfect way to patch without problems, it has become increasingly important to make router updates part of your patching routine. "It's not that you should panic. But it means that when a [router] vendor tells you to patch, then, yes, patch," Kaminsky says.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message