'Shift Left' Gets Pushback, Triggers Security Soul Searching

The common wisdom in the software industry is that fixing a vulnerability during production is 100 times more expensive than fixing it during the design phase. This massive purported cost of defects has fueled arguments — especially from vendors — that developers need increasingly complex — and expensive — tools to catch more bugs earlier in the development pipeline.

Yet software security professionals are now questioning the extreme nature of that financial trade-off.

In a draft report released last week, the Cybersecurity and Infrastructure Security Agency (CISA) noted that the origins of the factor-of-100 figure remain shrouded by four decades of rote repetition and that, even if true, the software development process that may have supported the figure has since changed. In short, agile development and the ability to push code to deployment rapidly and frequently may have reduced the cost of fixing mistakes in production code. This means the effort to saddle developers with the responsibility for code security — referred to as "shifting left" — may be overwrought.

Chris Hughes, CEO and co-founder of Aquia, a digital transformation security firm, didn't pull any punches in a LinkedIn post, using a vulgarity to describe shift left.

"Security beats Developers over the head with these poor quality noisy outputs, slowing down velocity and ultimately the business," Hughes said.

Other security and software experts weighed in on the LinkedIn post in a heated discussion — some in whole-hearted agreement, others challenging the notion that fixing software defects as early as possible is anything other than "common sense."

The kerfuffle is the latest sign of resurgent tensions between arguably a majority of developers, who see security requirements as a hurdle to better productivity, and DevSecOps-style developers and application security specialists, who see secure software as a quality target that also inevitably saves money.

Questioning the Common Wisdom

On Oct. 11, CISA published a report to its director on the Secure by Design initiative, an effort that aims to drive security into the software development and design phases to eliminate vulnerabilities that have allowed significant damage to critical networks and the compromise of sensitive information. The report noted specific challenges in convincing organizations to adopt better security practices, such as a lack of economic incentives for businesses to invest in security and to fix vulnerabilities. Companies such as Target and SolarWinds demonstrate that significant incidents do not lead to financial consequences, as both companies retained customers and recovered any lost market capitalization.

As a result, it remains unclear whether — and how much — companies should shift the security responsibilities leftward to developers, CISA stated in the report. Finding that balance for organizations is not a clear-cut effort.

"It is a commonly held belief that fixing vulnerabilities earlier is more cost effective," the report stated. "'Shift left' emphasizes moving testing activities earlier in the development process, with the notion that earlier identification of issues is better and produces a higher quality product. The challenge is in quantifying how much investment needs to be made."

Aquia's Hughes stressed in an interview with Dark Reading that the point of his post is that developers should be trained in security and offered better tools to secure products, but not by arguing with unsupported economic data.

"Businesses are focused on the financial aspect— they're motivated differently than security. As much as we wish that security was the only thing they cared about, it's simply not," Hughes said. "The need to worry about speed to market and feature velocity, rolling out new features and capabilities for customers. ... There's many benefits for shift left, but the financial benefit may not be one of them, and that [was] a big way to motivate the business from a financial perspective."

Not the Metrics You're Looking For...

The idea that bugs cost much more to fix in production systems than during the design stage started in the 1970s, as computer scientists and operations engineers studied software engineering. Barry Boehm, who served as chief scientist at TRW Defense Systems Group and as a distinguished professor of computer science and industrial engineering at the University of Southern California, created the Constructive Cost Model (COCOMO) of software engineering economics in the late 1970s and detailed its applications in his book, Software Engineering Economics, published in 1981.

In a 2021 paper, Boehm credited the 100x factor to a prior paper, "Industrial Metrics Top 10 List," which he published in 1987. Yet even Boehm noted that the measurement had likely changed over the years, saying that fixing a software problem after delivery is "often 100 times more expensive" and highlighting that the insertion of the word "often" was an update to his previous thinking.

"One insight shows the cost-escalation factor for small, noncritical software systems to be more like 5:1 than 100:1," Boehm stated in the 2001 paper "Software Defect Reduction Top 10 List." "This ratio reveals that we can develop such systems more efficiently in a less formal, continuous prototype mode that still emphasizes getting things right early rather than late."

Other data on the costs of fixing software defects included a 15:1 estimate calculated from detailed responses to a survey conducted by the National Institute of Standards and Technology (NIST), according to a 2002 report, "The Economic Impacts of Inadequate Infrastructure for Software Testing."

A survey of software developers finds that it takes 15 times more effort to fix a software defect after release compared to the requirements phase. Source: The Economic Impacts of Inadequate Infrastructure for Software Testing, NIST

The increasing focus on cloud-native and DevOps processes has led to a reduction in the cost of updating applications and, thus, the cost of distributing software fixes. The process of distributing tape, disks, or CDs with new software in the 1980s and 1990s has evolved into online updates and software-as-a-service, which requires no action on the part of the user and are much cheaper to update.

In one case study, a large health insurer implemented better defect detection and tracked the savings of fixing bugs earlier from 2013 to 2017 . It concluded that the company saved about $21 million from its previous annual security costs of $28 million. The case study, authored by then Aetna CISO Jim Routh and software security guru Gary McGraw, suggests that triaging bugs later costs four times more than fixing them during development.

"While the costs have absolutely changed, the final principle has not," says Routh, now the chief trust officer for cloud identity firm Saviynt. "It's still less expensive to produce quality software" than to produce buggy software and fix it later.

Adopting a culture of DevSecOps can help. Rather than forcing developers to use specific tools, application security specialists should work with them to develop a process for producing resilient code, says Routh.

Shift Left Still Makes Financial Sense

As CISA points out, the question that remains unanswered is how much the economics of software engineering say companies should focus on quality assurance, security, and resilience. A lot of assumptions need to be updated, and companies should be fostering a DevSecOps mentality, says Janet Worthington, senior analyst with business intelligence firm Forrester Research.

"When you say the phrase 'shift left,' I think it can imply to some people ... that it's just a set of tools that developers have to implement and all the burden is on them," she says. "And I think there's been a reaction over the years that you can't just put the burden on developers for security."

By embedding security knowledge throughout not only development but also testing and operations, companies create a more resilient foundation on which to build and deploy software, she says.

In the end, however, the question seems to be not whether fixing software earlier is better or more cost effective, but asking what needs to be better studied to determine how much to invest in driving security through development or operations.

Executives and DevOps teams need to take a total cost of ownership approach to development costs, says Gary McGraw, author of more than a half-dozen books on software security and former chief technical officer at Cigital, a software security firm.

"Developers should be deeply into securing their software," he says, adding that companies should have a software security specialist on every DevSecOps team who can participate, creating security features, doing security testing, and checking security design as a member of the team.

In his experience, there is no question that preventing problems now is better — from a quality, resilience, and security standpoint — than waiting until later.

"It's cheaper to fix bugs when you're still coding. It's cheaper to fix architecture when you're still thinking it up," he says. "Ultimately, the shift left thing is absolutely correct."