Why Don't Firewalls Work?

A large enterprise recently realized one of its firewalls had suddenly gone quiet, and was no longer requesting policy changes or updates as usual. After a careful audit, the enterprise discovered the reason: A hacker had inserted an "allow any to any" rule in the middle of the firewall's policy, leaving the doors wide open to all traffic, both in and out of the company.

Such blatant hacks are rare, but the story -- relayed by firewall maker Cisco Systems -- is a cautionary tale, experts say. A poorly configured firewall can be worse for data security than no firewall at all.

"We see customers that have hundreds of rules and thousands of objects defined on their firewalls," says Matt Dryer, product marketing manager for the Security Technology Business Unit at Cisco. "They go in and add rules and objects, but they never delete anything. They don't follow a structured change control process. They don't have an unmanageable amount of gear, but the configuration and change management process just keeps getting more complex."

The proliferation of firewalls in large enterprises is making management even more difficult, notes Mike Lloyd, chief scientist at RedSeal, which makes software that aids in enterprise firewall change and security posture management. "The problem with firewalls is that they were originally designed to secure a closed environment, like you'd secure a bank. But today's enterprise is more like a city than a bank. There needs to be some fundamental change in the way enterprises think about their firewalls."

The problem, experts say, is not in the firewall technology itself, but in the way the firewalls are administered. In most companies, firewall administrators have a wide variety of other responsibilities, and they simply don't have the time or information they need to set all of the rules properly.

"About 95 percent of firewall issues are configuration errors, not vulnerabilities in the firewalls themselves, says Nimrod Reichenberg, vice president of marketing at AlgoSec, which makes tools for firewall configuration and change management. "Most of the issues are caused by human error."

Mike Rothman, an analyst at security consulting firm Securosis, agrees. "Most of the issues with firewalls relate to the user opening something they shouldn't," he says. "This could be because a user asks for a port to be opened, and the admin doesn't realize what the impact of that is. Or it could involve adding [or removing] a rule, which obviates more stringent controls lower in the rule base. The bad guys are constantly doing reconnaissance to figure out which ports and protocols are open, and then attacking them. So if a perimeter firewall is inadvertently opened, there is a pretty high likelihood the issue will be discovered and exploited quickly."

For most companies, problems with the firewall emerge not because of a breach, but because of a compliance audit. Firewalls are a key element in many audits that involve compliance with SOX, PCI, or other regulations, and a misconfigured firewall is more likely to be discovered by an auditor than a hacker, experts say.

"The companies that are regulated generally have to do an audit on an annual basis," Dreyer observes. "That's when the issues usually come out."

In a perfect world, companies would audit their firewalls much more frequently to delete unnecessary rules and test the impact of rule changes, Dreyer says. But most IT organizations operate in a world that's far from perfect.

"In a typical enterprise, you'll see at least 10 change requests a day, going up to as many as 200," RedSeal's Lloyd says. "It's more than a human can handle."

Avishai Wool, CTO and co-founder of AlgoSec, concurs. He notes that when security staff turns over in organizations, much of the knowledge about how firewalls are configured is lost. Most IT organizations don't have time to do research on how and why firewall rules are set, and most of them don't have time to test new rules and changes to see what their impact might be on the overall security picture, he notes.

At the recent Computer Security Institute conference in Washington, D.C., Lloyd outlined a plethora of ways in which a single error in syntax in a single line of firewall code can create major security vulnerabilities. "What may seem arcane might be exactly the sort of thing that an attacker is looking for," he says.

In many cases, the errors occur not on the inbound side of the firewall, but on the outbound side, Lloyd observes. "They've written reasonable controls for traffic coming into the environment, but they write rules that allow just about anything out," he says. "The assumption is that outbound traffic is trusted, but they don't put enough thought into what might be exposed. A credit card database doesn't need to surf the Internet. But a lot of them are exposed to it."

Companies are also struggling with firewall sprawl, in which firewalls become more numerous and, in some cases, disparate. Mergers and acquisitions often create a mishmash of firewall technology, often making it difficult to manage and create policies.

"[Firewall sprawl] increases complexity, and complexity tends to result in human error," Rothman says. "Human error results in configuration errors, which will let attack traffic through. Yes, some of the firewalls do have light audit capabilities to make sure an admin doesn't do something totally stupid. But, ultimately, with a couple hundred devices to manage, errors are going to happen."

Many companies don't test their new firewall rules and configurations before they deploy them, and most do not have a sense of what risks they face if they make the wrong configuration decision, experts say.

"Most enterprises have some sort of 'change review board,' where the people who set the policies meet the people in operations to discuss the potential impact of a change," Lloyd notes. "But how many of these boards can really assess risk? Do they really understand the potential impact of poor configuration on the business? In most cases, they don't."

With so many configuration errors happening all the time -- and so much potentially sensitive data riding on those errors -- it's not surprising that a cottage industry is forming around the concept of security change and configuration management. Companies like AlgoSec, RedSeal, Tufin, and Skybox all are attempting to crack the enterprise firewall configuration equation, although some are more nuts-and-bolts operations tools while others offer a broader view of the status of these security systems in a single location, sometimes called security posture management (SPOM).

Rothman says the emerging generation of firewall configuration tools has been born because most of the management tools offered by the firewall vendors are short on functionality and often don't work in multivendor environments.

"The reality is the only reason there is a market for 3rd party firewall management tools is because the firewall vendors screwed it up," Rothman says. "There are lots of nice capabilities offered by these tools, especially for the big companies that have to manage hundreds of devices. The tools aren't cheap, but if screwing up the firewall config adds significant risk, or you can deploy people to do other activities, then it may be worth it."

Cisco's Dreyer obviously doesn't agree with Rothman's assessment of firewall vendors' tools, but he does agree there is real value in the third-party software that's emerging in the change and configuration management space. In fact, the company is planning a policy management "ecosystem" of development APIs in the coming year that will help third parties interface with Cisco's network and firewall management systems.

"The work that companies like Tufin, AlgoSec, and Skybox are doing on policy migration can be really helpful in companies that have mergers and acquisitions," Dreyer observes. "A company that does an acquisition and ends up migrating 1,400 firewalls needs some help with the policy life cycle."

Chris King, director of product marketing at next-generation firewall maker Palo Alto Networks, offers another alternative: moving away from current firewall technology and toward application-aware firewalls that might be easier to manage in the long run.

"I have seen enterprises that have 1,500 policies on a Check Point firewall," King says. "The way traditional firewalls are set up, you're always looking at what you need to deny. If you move to an application-based firewall, you might allow 100 apps, but you'd still have only 100 rules."

"Given that everything is encapsulated in a standard port, clearly a firewall that can only set policies based on ports/protocols is inherently limiting," Rothman says. "So on that point, we agree with Palo Alto Networks. The question is how you get there. Firewalls plus IPS-based application rules can be a decent interim step as opposed to rip and replace, though this approach will have scaling issues.

"But there is no question application awareness on the firewall is a critical capability today and will be more important tomorrow," he adds. "We expect all the firewall vendors to be moving in this direction. Firewall admins will need to evolve as well, since setting application policies is totally different than setting ports/protocols-based policies."

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.