Testing & Automation Pay Off for NSA's DevSecOps Project

For companies concerned with security as they push for faster development, a hallmark of agile frameworks such as DevOps, the US National Security Agency (NSA) has a message for you: Test, take it slow, but do it right and your developers will thank you.

Starting in 2018, the NSA embarked on a project to build more support for its developers and combine multiple instances of source code repositories. While the secretive government agency — whose missions include both spying on the communications of other countries and striving to secure US classified networks — has to work within the restrictions of a largely air-gapped network, the DevX team created a "highly available, elastically scalable architecture in AWS" that allows fast software development in a secure environment, Eric Mosher, technical lead for the DevX DevOps Pipeline at the NSA, said in a presentation at the virtual GitLab Commit conference last week.

Originally, installed from code in 2013, the NSA's source repository languished until 2018, when the agency kicked off its DevX project to improve the experience of developers. The team did an extensive rearchitecting of its approach to keep its security requirements while allowing the fewest custom changes as possible. Even with numerous dry runs, the developers still ran into problems, Mosher said.

"We didn't know the right kind of testing to do, and we needed some smarter automation," he said. "After the upgrade, we made sure to build in more of this automated testing ... and we added that smarter automation."

In the end, however, the project resulted in a fully DevOps-capable pipeline that allows government developers to build features more quickly yet still comply with the strict security regulations governing the NSA's operation, Mosher said.

"One of the things that is really great for me is having people coming to us and say thank you ... you are really empowering our mission," he said.

The NSA joins many other companies in committing to DevOps, and the security-inclusive DevSecOps, which have both made significant inroads into companies in the past year. No wonder: Developers get measured on the quality of their code (36%) and using feedback from customers (35%), and security plays into both of those metrics, Amy DeMartine, vice president and research director with Forrester Research said during a presentation at the virtual conference.

"What do developers care about? Developers are measured quite differently than the security team, [but] when you think about what they are looking at and how they are measured and what their success looks like, security plays into both of those," she said.

While business competition is pushing companies to speed up their development, the realities of cybersecurity are forcing companies to incorporate more security checks into development. In 2019, for example, a third of companies suffered a breach, according to Forrester, and for external attacks, the top causes of breaches were software vulnerabilities, accounting for 40% of specified causes, and web application flaws, accounting for 37%. (The survey allowed more than one answer.)

Automation, however, can both speed development and improve security. While developers' top priorities in the next 12 months include using more cloud-based development environments and better meeting their company's business demands, automation features heavily in the top 10 future priorities. Developers want to increase automation of secure development life cycle, speed up their release and deployment cycles, and use more open source software to decrease development time. 

The result is that 38% of developers are releasing software monthly or faster, up from 27% in 2018, according to Forrester Research.

The same automation applied to security makes a difference, Forrester's DeMartine said. Companies that automatically scan their applications on a daily basis — 300 times or more a year — fix vulnerabilities 11.5 times faster than those who scan less than three times a year, according to an analysis by software-security firm Veracode and cited by DeMartine.

"They are trying to release faster, they are trying to get their genius out to customers faster than they ever had before," she said. 

Forrester recommends that companies implement static application security testing (SAST), dynamic application security testing (DAST), and software composition analysis (SCA). So far, only a third of companies have implemented SAST at the development stage, but more companies plan to implement DAST during development and testing, — 34% and 43%, respectively — with the more interactive application security testing (IAST) showing even better numbers. 

Software composition analysis, which includes managing and checking the security of open source components is also taking off with 37% of companies planning to add the capability to development, up from 31% who have it implemented today, according to Forrester. 

In the end, the NSA widely adopted automation to speed up — and make more reliable — its own infrastructure. The agency uses Terraform for cloud-infrastructure automation, Ansible for build automation, and GitLab for continuous integration and continuous deployment. Out of the large infrastructure created in the cloud, only two applications are custom to the NSA, Mosher said.

"That's why we are able to move quickly," he said. "We cut down on any custom work that we are doing, and those custom things are very, very stable and they integrate well."