When AI Becomes the Hacker

For the past couple of years, renowned technologist and researcher Bruce Schneier has been researching how societal systems can be hacked, specifically the rules of financial markets, laws, and the tax code. That led him to his latest examination of the potential unintended consequences of artificial intelligence on society: how AI systems themselves, which he refers to as "AIs," could evolve such that they automatically - and inadvertently - actually abuse societal systems.

"It's AIs as the hacker," he says, rather than hackers hacking AI systems.

Schneier will discuss his AI hacker research in a keynote address on Monday at the 2021 RSA Conference, which, due to the pandemic, is being held online rather than in person in San Francisco. The AI topic is based on a recent essay he wrote for the Cyber Project and Council for the Responsible Use of AI at the Belfer Center for Science and International Affairs at Harvard Kennedy School.

The core question Schneier asks is this: What if artificial intelligence systems could hack social, economic, and political systems at the computer scale, speed, and range such that humans couldn't detect it in time and suffered the consequences?

It's where AIs evolve into "the creative process of finding hacks."

"They're already doing that in software, finding vulnerabilities in computer code. They're not that good at it, but eventually they will get better [while] humans stay the same" in their vulnerability discovery capabilities, he says. 

In less than a decade from now, Schneier predicts, AIs will be able to "beat" humans in capture-the-flag hacking contests, pointing to the DEFCON contest in 2016 when an AI-only team called Mayhem came in dead last against all-human teams. That's because AI technology will evolve and surpass human capability.

Schneier says it's not so much AIs "breaking into" systems, but AIs creating their own solutions. "AI comes up with a hack and a vulnerability, and then humans look at it and say, 'That's good,'" and use it as a way to make money, like with hedge funds in the financial sector, he says.

The irony here, of course, is that AI starts with human input and programming. Frankenstein analogies aside, the core problem is that AI doesn't have the same human cognitive functions like empathy or a gut check to know where to draw the line. Schneier notes that while there's plenty of research on incorporating context, ethics, and values into AI programs, it's not a built-in function of today's AI systems.

Even so, he says, humans will employ AI to find loopholes in tax codes, such as a major accounting firm doing so to find a new "tax dodge to sell to their customers." So financial firms aren't likely to "program in" rules that thwart their ability to monetize AI knowledge.

The biggest risk is that AIs will find a way around a rule without humans knowing - "that AIs will figure out something that will hack the rules and we won't realize that," Schneier says.

Schneier points to the Volkswagen scandal in 2015, when the carmaker was caught cheating on emissions control-level tests of its vehicle models after engineers programmed the cars' computer systems to activate emissions-curbing only during tests, not in its normal operations. 

"There it was humans attacking the rules" and not AI itself, he says, but it's a good example of what AI could ultimately do to cheat a system if left unchecked to learn ways around it.

In his essay, "The Coming AI Hackers," Schneier describes it this way: "If I asked you to design a car's engine control software to maximize performance while still passing emissions control tests, you wouldn't design the software to cheat without understanding that you were cheating. This simply isn't true for an AI; it doesn't understand the abstract concept of cheating. It will think 'out of the box' simply because it won't have a conception of the box, or of the limitations of existing human solutions. Or of ethics. It won't understand that the Volkswagen solution harms others, that it undermines the intent of the emissions control tests, or that it is breaking the law."

A Wake-up Call to Action
Schneier admits the concept of AIs as hackers is "super speculative" for now, but it's an issue that needs to be addressed. 

"We need to think about this," he says. "And I'm not sure you can stop this. The ease of this [AIs hacking] happening depends a lot on the domain [in question]: How can we codify the rules of the system?"

The key is harnessing AIs for defense, like finding and fixing all vulnerabilities in a software program before it gets released. 

"We'd then live in a world where software vulnerabilities were a thing of the past," he says. 

The downside is the transition period would be vulnerable: Legacy or already-released code could be at risk of attack by AI tools abused by adversaries, he says.

The risk is AI systems hacking other AI systems in the future, and humans experiencing the fallout, he says.

Schneier's latest AI research evolved out of his study of how the hacker mindset and skills could be applied to securing societal systems, which he first presented at the 2020 RSA Conference in San Francisco. This concept, which he coined "hacking society," would mean ethical hackers helping fix the US tax code and legislation to avoid inadvertent or deliberate loopholes, for example.

His big idea boils down to this: "Can we hack society and help secure the systems that make up society?"

Meanwhile, keep an eye on AIs hacking society. 

"Computers are much faster than people. A human process that might take months or years could get compressed to days, hours, or even seconds. What might happen when you feed an AI the entire US tax code and command it to figure out all of the ways one can minimize the amount of tax owed?" he wrote in his essay.