To Spot Attacks Through AI Models, Companies Need Visibility

As companies rush to develop and test artificial intelligence and machine learning (AI/ML) models in their products and daily operations, the security of the models is often an afterthought, putting the firms at risk of falling prey to backdoor and hijacked models.

Companies with their own ML team have more than 1,600 models in production, and 61% of companies acknowledge that they do not have good visibility into all of their ML assets, according to survey data published by HiddenLayer, an AI/ML security firm. The result: Attackers have identified models as a potential vector for compromising companies, with a recent exploration by software security firm JFrog into models posted to the Hugging Face repository finding malicious files that create a backdoor on the victim's machine.

Companies need to look at the security of the AI/ML models and their MLOps pipeline as they rush to develop AI-enabled capabilities, says Eoin Wickens, technical research director at HiddenLayer.

"With the democratization of AI and the ease with which pretrained models can be downloaded from model repositories these days, you can get a model, fine-tune it for purpose, and put it into production easier now than ever," he says. "It remains an open question as to how we can ensure the safety and security of these models once they've been deployed."

The pace of AI adoption has security experts concerned. In a talk at Black Hat Asia in April, two security researchers with Dropbox will present their investigation into how malicious models can attack the environments in which they are executed. The research identified ways of hijacking models, where running the model allows embedded malware to compromise the host environment, and backdooring, where the model has been modified to influence its behavior and produce certain outcomes.

Without efforts to check the security and integrity of models, attackers could easily find ways to run code or bias the resulting output, says Adrian Wood, a security engineer with the red team at Dropbox and a co-presenter at Black Hat Asia.

Data scientists and AI developers are "using models from repositories that are made by all kinds of people and all kinds of organizations, and they are grabbing and running those models," he says. "The problem is they are just programs, and any program can contain anything, so when they run it, it can cause all sorts of problems."

The Fog of AI Models

The estimate of more than 1,600 AI models in production may sound high, but companies with teams focused on data science, ML, or data-focused AI have a lot of models in production, says Tom Bonner, vice president of research at HiddenLayer. Over a year ago, when the company's red team conducted a pre-engagement assessment of a financial services organization, they only expected a handful of ML and AI models to be in production. The real number? More than 500, he says.

"We're starting to see that, with a lot of places, they're training up perhaps small models for very specific tasks, but obviously that counts to the sort of overall AI ecosystem at the end of the day," Bonner says. "So whether it's finance, cybersecurity, or payment processes [that they are applying AI to], we're starting to see a huge uptick in the number of models people are training themselves in-house."

Companies' lack of visibility into what models have been downloaded by data scientists and ML application developers means that they no longer have control over their AI attack surface.

Pickle, Keras: Easy to Insert Malware

Models are frequently created using frameworks, all of which save model data in file formats that are able to execute code on an unwary data scientist's machine. Popular frameworks include TensorFlow, PyTorch, Scikit-Learn, and, to a lesser degree, Keras, which is built on top of TensorFlow. In their rush to adopt generative AI, many companies are also downloading pretrained models from sites such as Hugging Face, Tensorflow Hub, PyTorch Hub, and Model Zoo. 

Typically, models are saved as Pickle files by Scikit-Learn (.pkl) and PyTorch (.pt), and as the Hierarchical Data Format version 5 (HDF5) files often used by Keras and TensorFlow. Unfortunately, these file formats can contain executable code and often have insecure serialization functions that are prone to vulnerabilities. In both cases, an attacker could attack the machines on which the model is run, says Diana Kelley, chief information security officer at Protect AI, an AI application security firm. 

"Because of the way that models work, they tend to run with very high privilege within an organization, so they have a lot of access to things because they have to touch or get input from data sources," she says. "So if you can put something malicious into a model, then that would be a very viable attack."

Hugging Face, for example, now boasts more than 540,000 models, up from less than 100,000 at the end of 2022. Protect AI scanned Hugging Face and found 3,354 unsafe models — about 1,350 that were missed by Hugging Face's own scanner, the company stated in January.

Companies Need Ability to Trust Training Data

To secure their development and deployment of AI models, organizations should integrate security throughout the ML pipeline, a concept often referred to as MLSecOps, experts say.

That visibility should start with the training data used to create models. Making sure that the models are trained on high-quality and secured data that cannot be changed by a malicious source, for example, is critical to the ability to trust the final AI/ML system. In a paper published last year, a team of researchers, including Google DeepMind engineer Nicholas Carlini, found that attackers could easily poison the training of AI models by buying up domains that were known to be included in the data sets. 

The team responsible for the security of the ML pipeline should know every source of data used to create a specific model, says Hidden Layer's Wickens.

"You need to understand your ML operations life cycle, from your data-gathering and data-curating process to feature engineering — all the way through to model creation and deployment," he says. "The data you use may be fallible."

Scoring Models for Security

Companies can start with looking at metrics that can hint at the underlying security of the model. Similarly to the open source software world, where companies are increasingly using tools that use different open source project attributes to create a report card for security, available information about a model can hint at its underlying security. 

Trusting downloaded models can be difficult as many are made by ML researchers who may have little in the way of a track record. HiddenLayer's ModelScanner, for example, analyzes models from public repositories and scans them for malicious code. Automated tools, such as Radar from Protect AI, produce a list of the bills of materials used in an AI pipeline and then determine whether any of the sources pose a risk. 

Companies need to quickly implement an ecosystem of security tools around ML components in much the same way as the open source projects have created security capabilities for that ecosystem, says Protect AI's Kelley.

"All those lessons we learned about securing open source and using open source responsibly and safely are going to be very valuable as the entire technical planet continues the journey of adopting AI and ML," she says. 

Overall, companies should start with gaining more visibility into their pipeline. Without that knowledge, it's hard to prevent model-based attacks, Kelley says.