Malicious attack method on hosted ML models now targets PyPI

Artificial intelligence (AI) and machine learning (ML) are now inextricably linked to the software supply chain. ML models, which are based on large language models (LLMs), are powering the enterprise — and offer an infinite number of solutions to organizations’ mission-critical needs. The widespread and increasing use of generative AI tools like OpenAI’s ChatGPT, in addition to developer community resources like Hugging Face – a platform dedicated to collaboration and sharing of ML projects – show how software, coding and AI/ML are now one and the same.

But as with any new technological advancement, the pressing need for ML models has created a new and ever-evolving attack surface that the cybersecurity industry is racing to understand and mitigate. Recognizing the convergence of AI and the software supply chain, ReversingLabs (RL) researchers and engineers have taken steps to better understand the threat posed by malicious ML models.

One such threat that RL researchers have previously flagged is the Pickle file format, a popular but insecure Python module that is used widely for serializing and deserializing ML model data. Dhaval Shah, RL’s senior director of product management, wrote recently that Pickle files open the door to malicious actors who can abuse it to inject harmful code into the model files.

That warning proved true with the discovery of nullifAI, discovered by RL threat researchers in February, in which threat actors abused ML models in the Pickle file format to distribute malicious ML models on Hugging Face. With this latest discovery, RL researchers uncovered a new malicious campaign that further proves threat actors’ newly favored method of exploiting the Pickle file format — this time on the Python Package Index (PyPI).

Last Tuesday, RL researchers detected three, newly uploaded malicious packages that pose as a “Python SDK for interacting with Aliyun AI Labs services.” As the package description indicates, this is an attack that targets users of Alibaba AI labs. Once installed, the malicious package delivers an infostealer payload hidden inside a PyTorch model loaded from the initialization script. (PyTorch models are basically zipped Pickle files.) The malicious payload exfiltrates basic information about the infected machine and the content of the .gitconfig file.

Here’s what RL researchers discovered — and what this new malicious campaign means for the security of ML models, as well as how open-source software (OSS) platforms are still a favored supply chain attack vector.

Alibaba AI used as a lure

The malicious PyPI packages in question are aliyun-ai-labs-snippets-sdk, ai-labs-snippets-sdk and aliyun-ai-labs-sdk. They present themselves as AI Labs SDK (software development kit) for Python. However, the packages have no connection to AI Labs and contain no SDK functionality. They are simply designed to exfiltrate reconnaissance information back to an attacker-controlled server.

Figure 1: The readme page for one of the malicious PyPI packages

The scenario in which potential targets would decide to use these packages is not clear, but the infection vector likely includes a supporting phishing or social engineering campaign. The malicious packages were published to PyPI on May 19 and were available for download for less than 24 hours. RL estimates that the three packages were collectively downloaded about 1,600 times. The ai-labs-snippets-sdk package accounted for the majority of downloads, due to it being available for download longer than the other two packages.

Malicious ML models hidden in PyTorch

Regardless of the infection vector, the most interesting aspect of this campaign is that the payload is hidden inside a malicious PyTorch model located in two of the three packages.

RL Spectra Assure enhancements has resulted in improved support for the identification and unpacking of ML file formats, as well as the implementation of new Threat Hunting Policies (THPs), which are designated to pinpoint and explain risks related to the presence of dangerous functions inside ML model files.

Those platform improvements were recently demonstrated when RL threat researchers detected several malicious ML models hosted on the Hugging Face platform in the nullifAI campaign. While malicious models from that campaign weren’t fully functional, they presented the risks related to the inclusion of models from untrusted sources into an organization’s development environment.

In this new campaign, the models found in the new malicious PyPI packages contain fully functional infostealer code. Why would malware authors hide code in ML models that are Pickle formatted files? Most likely because security tools are just starting to implement support for detection of malicious behavior in ML file formats, which have been traditionally viewed as a medium for sharing data, not distributing executable code. This real-world example demonstrates why it is important to set up a zero-trust boundary for different types of files that are incorporated into your development environment.

How the new ML compromise works

In these latest examples of AI-focused supply chain attacks, the malicious PyTorch models are loaded from the __init__.py script immediately upon installation, as visible in Figure 2.

Figure 2: Initialization script responsible for loading of the malicious PyTorch model

The malicious models contain Python code that is designed to steal information about the logged user, the network address of the infected machine, the name of the organization that the machine belongs to, and the content of the .gitconfig file (Figure 3). The name of the organization is retrieved by reading the _utmc_lui_ preference key from the configuration of the AliMeeting online meeting application, an alternative for video-conferencing applications like Zoom that is popular in China. That clue, combined with the fact that the content of .gitconfig file is being exfiltrated, are strong indications that the likely targets of this campaign are developers located in China.

Figure 3: Infostealer payload extracted from PyTorch model

In some versions of the packages, the malicious payload from the PyTorch model is obfuscated by an additional layer of a Base64 encoding, making it even harder to detect.

To detect malicious models, you need modern tooling

RL’s recent enhancements to the identification and unpacking of different ML model file formats, and the implementation of the newest THP, made this latest discovery possible. Figure 4 shows the THPs that were triggered by the packages containing models with the malicious payload.

Figure 4: Triggered THPs for the packages with malicious ML model

These include TH19101 and TH19103, which warn about serialized data formats like Pickle files that are capable of creating a new process and executing code — something you wouldn’t expect from pure, serialized data.

Figure 5 shows triggered THPs for the models that include a flag for the added layer of Base64 obfuscation. In this case, the main difference is the triggering of TH16103, which warns of the presence of files that can dynamically execute Base-encoded data, suggesting the presence of the mentioned Base64 encoding obfuscation layer.

Figure 5: Triggered THPs for the packages with additional Base64 obfuscation layer

Conclusion

Malicious PyPI packages are not uncommon. RL threat researchers encounter them on a weekly – even a daily — basis. But threat actors are always trying to find new ways to hide the malicious payloads from security tools — and security analysts. This time, they were using ML models, a novel approach for distribution of malware via the PyPI platform.

This is a clever approach, since security tools are only starting to implement support for the detection of malicious functionality inside ML models. Reporting security risks related to ML model file formats is also in its early stages. To put it simply, security tools are at a primitive level when it comes to malicious ML model detection. Legacy security tooling is currently lacking this required functionality.

Recognizing the critical nature of malicious ML model detection for software supply chain security, RL has been proactive in this field. Spectra Assure, RL’s software supply chain security solution, currently provides support in the detection of security threats related to ML models, and is able to generate an ML-BOM, which is based on the CycloneDX standard for bills of materials and provides immediate visibility into ML model in an organization’s environment.

Indicators of Compromise (IOCs)

Indicators of Compromise (IoCs) refer to forensic artifacts or evidence related to a security breach or unauthorized activity on a computer network or system. IOCs play a crucial role in cybersecurity investigations and incident response efforts, helping analysts and security professionals identify and detect potential security incidents.

The following IOCs were collected as part of RL’s investigation of this malicious software supply chain campaign.