Software Patches Eat Government IT's Lunch

Netscape co-founder and prominent tech investor Marc Andreessen famously noted that "software is eating the world." Unfortunately, it's also eating the lunch of most enterprises, including federal agencies.

For all the talk about wasteful government IT spending, little is said about the costs agencies pay to patch buggy software, a consequence of the industry's predisposition to release their wares now and fix them later. For Robert Jack, CIO of the U.S. Marine Corps, those costs aren't incidental.

"We have roughly 300,000 people, of which a third have day-to-day access to the enterprise network," Jack said at a recent forum on cybersecurity. "I have to defend the network at the desktop or end-user device. I have over 450 registered systems that are regressed to 10 significant versions. When we get a patch from a vendor, we have to go out and test that against all that."

He continued, "Think about the labor hours where I have to touch [and administer patches on] all those devices. And that's just for one patch." Every week, dozens of new vulnerabilities are catalogued by US-CERT, the government's computer emergency readiness team, offering a glimpse of the headaches Jack and CIOs like him face.

Speaking to the software industry at large, Jack said bluntly, "You're killing me."

[ As cloud and mobile proliferates, federal IT leaders should take more data-centric approach to security. Read Secure Data, Not Devices. ]

The staggering cost of software bugs is hard to nail down. However, a Cambridge University study released earlier this year estimated that finding and fixing coding problems costs software makers and the global economy $312 billion a year. That doesn't reflect what customers must also spend to patch and maintain that software on their networks.

The problem, however, goes well beyond the mechanics of software and system maintenance. It also goes to the heart of network security and the growing risks associated with unknown software vulnerabilities, Jack said. Having spent 40 years in charge of command, control, communications, computers and cyber operations for the Air Force, the Defense Department and now the Marine Corps, Jack knows the problems as well as anyone.

Software by its nature is a work in progress. While vendors can't anticipate every problem, some of which are spawned when software interacts with other software on a network, vendors are making too many calculated compromises in order to ram their products and updates into production, Jack said. But worse, they're exposing organizations and their executives to growing liabilities if something goes wrong.

Jack pointed to recent reports, which he didn't specify, indicating that 25% of hospital operating room liability lawsuits are now tied to software coding problems. Lawsuits based on software failures are also becoming a big concern for the auto industry, he said, and the issue has prompted high-level discussions within the Defense Department.

It's only a matter of time before the high-profile enterprises become targets for liability lawyers looking to exploit software mishaps, Jack warned, adding that those in positions of authority ought to consider "looking for some big-time insurance." In a recent article on the growing threat of software product liability for the Berkeley Technology Law Journal, Lawrence Levy and Suzanne Bell noted, "As society increasingly relies on software to perform critical functions in everything from manufacturing to life-support systems, the risk that an error in a software program will lead to economic loss, property damage or personal injury increases."

One of the big questions surrounding software liability, however, is whether computer software is a good or a service. That's important, Levy and Bell say, because "the sales of goods, but not of services, are subject to the damages and warranty provisions of the Uniform Commercial Code." The courts, however, are now beginning to consider cases involving not only the software itself, but also significant maintenance and support services, and this is likely to impact more and more organizations.

In the meantime, Jack concluded, software vendors aren't likely to change the way they develop, test and deploy their products. "I've been beating that drum for 15 years," he said. "I don't believe legislating software assurance is going to work. I need corporate citizenry to step up to the plate and take responsibility for what they put into their software."

About the only thing government agencies can do is manage their risks. The fast pace of software adoption has all but rendered the government's approach to software security certification and accreditation obsolete. In fact, "the old certification and accreditation process has been gone for three years now," said Ron Ross, a senior security official at the National Institute of Standards and Technology, during the same forum.

NIST, which sets the security standards for government agency information systems, has moved to a risk management framework that calls on agencies to perform real-time network monitoring to identify attempts to exploit hardware and software vulnerabilities. About 10% of attacks will get through defenses no matter what, Ross said.

If you know that your system can withstand a cyber-attack and that malware can't spread through the network and bring you down, he added, authorizing officials should be in a better position to accept a certain degree of risk. However, most federal agencies' inability to replace legacy systems due to lack of funding and cultural inertia makes it difficult to manage all the risks associated with so much software.

The challenge is only getting greater for CIOs like Jack as government agencies expand their networks into the cloud and extend their services to mobile devices. While those moves hold the promise of new and greater efficiencies, they also add more layers of software and the inevitability of more software patching.