Boeing's DEF CON Debut a Sign of the Times

There was no flight simulator to fly a US Air Force F-35 fighter jet, no aircraft control-system circuit boards to tinker with at this year's Aerospace Village at DEF CON, which ran as an online event due to the COVID-19 pandemic. But unlike the in-person event in Las Vegas in 2019, when major airplane manufacturers Airbus and Boeing were not involved, this year Boeing joined the Village and signaled that it's ready to engage more closely with the hacker community.

Last year, the first-ever aviation-themed Village at DEF CON opened amid the backdrop of a heated dispute between Boeing and IOActive over researcher Ruben Santamarta's findings of security vulnerabilities in the on-board network of a Boeing 787 airplane. IOActive's Santamarta — who had presented his research over at Black Hat USA in Las Vegas just a few days before DEF CON kicked off — maintained that an attacker exploiting the flaws could remotely gain access to the aircraft's sensitive avionics network, also known as the crew information systems network.

Santamarta — who in 2018 shared research at Black Hat USA on how he was able to hack into in-flight airplane Wi-Fi networks and satellite communications equipment from the ground — in his newest research found that a piece of firmware in a core network component of the 787's network contained a menu of vulnerabilities, including buffer overflow, memory corruption, stack overflows, and denial-of-service flaws that he said an attacker could exploit remotely to ultimately reach the crew-information systems network module.

Boeing, based on its own internal testing of Santamarta's findings, argued that the vulnerabilities could not be exploited to affect a critical system on the plane nor could they be abused remotely to hack the avionics system. "After working with IOActive to understand its research, Boeing and its partners tested their findings in integrated environments, both in labs and on an airplane. Our extensive testing confirmed that existing defenses in the broader 787 network prevent the scenarios claimed," company spokesperson said at the time.

Another statement from Boeing got heated: "IOActive chose to ignore our verified results and limitations in its research, and instead made provocative statements as if they had access to and analyzed the working system. While we appreciate responsible engagement from independent cybersecurity researchers, we're disappointed in IOActive's irresponsible presentation," the company said at the time. 

IOActive, meanwhile, stood by Santamarta's research, noting that the company had been working with Boeing prior to the Black Hat presentation and that Boeing had "mischaracterized" Santamarta's findings. Santamarta, who conducted his research in a lab setting, indeed was careful to note in his findings and in his presentation that the ultimate effect on the actual avionics system was unclear without him having access to an actual 787 aircraft. But he said an attacker exploiting the firmware indeed could bypass security controls on the network and reach the avionics network. What the attacker could do from there was unclear, according to Santamarta.

One year later, Boeing and IOActive remain at a stalemate on the research, with neither budging on their conclusions. Even so, at DEF CON Boeing provided some insight into its thinking and lessons learned from the dispute, and there may be a silver lining for aviation cybersecurity now as well: Boeing says it's ready to "embrace" the security researcher community.

John Craig, chief engineer of cabin network and security at Boeing, calls last year's experience with IOActive a turning point for Boeing. The public clash between Boeing and IOActive led the aerospace vendor to beef up its vulnerability disclosure process and to create a so-called tech council that includes invited third-party security researchers that Boeing plans to bring into its internal labs post-pandemic to get hands-on experience with Boeing's airplane networking equipment and systems.

"We've turned a corner. We have come to the realization that we really need to embrace these security researchers because they have a really valuable perspective on our industry that we probably don't see all the time," Craig told Dark Reading in an interview last week.

Boeing had even planned to host a capture-the-flag contest with some of its hardware at DEF CON until the physical show was moved online, according to Craig. The Boeing exec was part of a panel about connecting the aviation ecosystem, along with officials from the Aviation-ISAC, Federal Aviation Administration, Department of Defense, and US Department of Homeland Security. Boeing also served as a sponsor of the Village, and a Boeing product engineer also gave a presentation on the secure development life cycle of an airplane.

Craig says he and some Boeing colleagues paid a visit the 2019 Aviation Village at DEF CON — and came away pleasantly surprised. "It was actually a very positive experience," he says. "It wasn't nefarious," he says of the DEF CON Village's activities. "People really were positive and wanted to make a difference. We loved the event."

Same Story, Different Chapter
Boeing's initial wariness of the hacker community, punctuated by its dispute with IOActive, is reminiscent of a long tradition of a culture gap between industries and security researchers that dates back to the early 2000s with Microsoft squaring off against researchers finding vulns in Windows. Since then, the automotive, medical device, Internet of Things, and industrial control systems (ICS) industries have ridden an often-painful learning curve after misconstruing the work that security researchers do when they root out holes in their products.

Aviation long has prioritized safety for obvious reasons. But it's no longer possible nor useful to either ignore or hide cybersecurity's role in airline safety as aircraft such as the 787 become more networked, experts say.

Pete Cooper, lead of the Aerospace Village and a senior fellow with the Atlantic Council, says the process of understanding and then working alongside ethical hackers is a journey for many organizations. "A lot of people get really nervous about the topic ... and are very protective" and don't want to address security of their industrial products, he says. "But we absolutely have to discuss it: That's where we find our way to move forward."

It's a two-way street: Researchers are often hesitant to approach the vendors for legal or other reasons. "It's [also] making the researchers more comfortable connecting with these guys, and we can make that [connection] from the Aviation-ISAC side," says Randy Talley, senior adviser for the Department of Homeland Security's Cybersecurity Infrastructure Security Agency. He points to Boeing's sponsorship participation in the Aerospace Village this year as a big step there.

Boeing also participated in the RSA Conference's much-smaller Aerospace Village in February, which was its first such security appearance and a "dry run" for DEF CON, according to Craig.

For Santamarta, Boeing's high-profile role at DEF CON and its newfound relationship with the hacker community falls flat after his unpleasant experience with Boeing in the disclosure process and his presentation. "I've been told that my research triggered a lot of initiatives around cybersecurity in the aviation sector, even in regulatory terms. That's good," Santamarta says. "What I find sad is that nobody dares to publicly reference the research" itself, he adds.

Craig says Boeing does not dispute issues in the code that Santamarta uncovered. Boeing tested the findings for months in its labs, he says, plus two days of testing the systems on an actual 787 aircraft. "They didn't really react the way he predicted they would," he says of Santamarta's research.

But he wouldn't elaborate on the technical details. "There are a lot of things that went into that that I really don't want to go into here. That's kind of where we ended it."

Alan Burke, associate deputy director of the Air Force Cyberspace Operations and Warfighter Communications, says Boeing briefed officials on their internal investigation and analysis of Santamarta's research. "We walked away impressed with the rigor Boeing put forward to address the researcher's claim and to put it to bed," says Burke, who participated in the aviation ecosystem panel at DEF CON.

Santamarta notes that although IOActive requested information about the actual version of the firmware Boeing used in its testing, Boeing never provided it to the company, he says.

"The focus has been moved to PR, but regarding the actual research there is cloak of silence that covers it all," Santamarta says. "I'd say that the only reason they don't publicly dispute the issues with the code is because everybody at [Black Hat] could see the code in the slides. Code doesn't lie."

And according to John Sheehy, senior vice president of research and strategy at IOActive, no one from IOActive has been invited by Boeing to participate in its technical advisory council.

"Nonetheless, we are encouraged to see Boeing putting a greater emphasis on product cybersecurity even during a financially challenging period for the company. We do believe Boeing has a fresh perspective on product cybersecurity and constructive engagement with the cybersecurity research community as a result of our interactions related to Ruben’s most recent research," Sheehy says. "Ultimately, we reached different technical conclusions than Boeing as part of Ruben's research project."

Sheehy says IOActive shared its research on the 787 publicly to provide insight to the aviation industry on product cybersecurity issues. "That project has been concluded and Ruben is working on his next project," he says.

As for the firmware flaws Santamarta found, Boeing is just now working with Honeywell to fix the vulnerabilities in the VxWorks 6.2-based system from Honeywell that operated as the 787's Crew Information System File Server/Maintenance System Module. Craig says that firmware update had been placed on a schedule at Boeing that prioritizes patching based on risk. "In the past, there were protections [for the vulnerabilities], and those protocols aren't adequate [now] so we're taking action" to fix it, he says.

And Craig acknowledged that Boeing's initial public response to Santamarta's research didn't come across well. On the DEF CON panel on Friday, he noted that "it wasn't really the intent, but I think it was viewed as hostile."

How Boeing Vets Third-Party Software
Boeing says it has a formal process for ensuring that third-party equipment and software on its aircraft are free of security vulnerabilities or potential weaknesses that could be exploited. According to Craig, the company is currently fine-tuning its security requirements for suppliers. "We are getting much firmer on security requirements and how they are deployed to external suppliers," he says. There's a similar process for internal code development as well.

Third-party testers conduct select static- and dynamic code analyses, he says, and Boeing also runs tabletop exercises on factory cybersecurity. The company also conducts penetration tests and vets any network changes. "Every network change on airplane gets a thorough test by design, and a third party comes out and validates it," Craig says. Security is embedded into the "entire build cycle for the airplane," he says.

Software updates for the on-board airplane network occur on average just once a year. "We also have configuration files that allow us to do a lot of things to help mitigate different aspects of the network" from threats, he says. "As we update the software, we make it available to customer airlines to incorporate."

Meanwhile, supply chain security in the aviation industry has the attention of the Federal Aviation Administration (FAA). The FAA currently is working on new policies for securing the aviation supply chain, according to Siddharth Gejji, manager of information security and privacy for the FAA. "Supply chain security is certainly a big issue; it's one of the biggest cybersecurity risks," Gejji says. "We work with the industry pretty closely" on this issue, he adds.