Malicious ML models found on Hugging Face Hub
Researchers have spotted two machine learning (ML) models containing malicious code on Hugging Face Hub, the popular online repository for datasets and pre-trained models.
Once one of them is downloaded and executed on the developer’s machine, the malicious payload checks if it is being executed on a Windows, Linux or a system using the Mach kernel (e.g., macOS). Depending on the results of the check, it uses different programming logic to create a reverse shell that connects to a hardcoded IP address.
The discovery
The Hugging Face Hub is an online platform where software developers and researchers can find, share and collaborate on ML models. These models provide functions that can be embedded in software applications.
A malicious model on Hugging Face Hub, “cleared” by security scans (Source: ReversingLabs)
ML models – i.e., their mathematical representations – can be stored in various data serialization formats. Among those is one called pickle.
“Pickle is a popular Python module that many teams use for serializing and deserializing ML model data. While easy to use, pickle is considered an unsafe data format, as it allows Python code to be executed during ML model deserialization [aka ‘unpickling’],” Reversing Labs researchers noted.
The two malicious models flagged by the researchers were stored in PyTorch format, which uses the pickle module for serialization and deserialization, making it effectively a compressed pickle file with a PyTorch wrapper.
ReversingLabs reverse engineer Karlo Zanki found that while the models are in PyTorch format, they have been zipped in 7z format instead of the default ZIP format, which prevents them from being loaded using PyTorch’s default torch.load() function.
“That is likely the reason why Picklescan — the tool used by the Hugging Face to detect suspicious Pickle files — did not flag them as unsafe,” Zanki pointed out.
Picklescan was thwarted by specially crafted “broken” Pickle files, they found: “The Picklescan tool showed an error when it encountered the inserted opcode that breaks the serialized stream, but failed to detect the presence of dangerous functions, even though they are invoked before the encountered stream compromise.”
This defense evasion technique, which the researchers nameed “NullifAI”, effectively exploits the execution of serialized code in pickle files to allow malicious code execution.
Counteracting new threats
The Hugging Face security team has removed the malicious models and has improved Picklescan’s detection of broken pickle files. Reversing Labs has published indicators of compromise. Judging by the screenshots of the (now deleted) repositories hosting the malicious models, they went on unnoticed for more than eight months.
“While Pickling is not using common evasion techniques like typosquatting, or attempting to mimic a popular, legitimate ML model, the packages look more like a proof-of-concept model for testing a novel attack method,” Zanki noted.
The name of one of the malicious models (“ballr7”) is similar to others previously discovered by JFrog’s researchers.
There are indications that point to researchers being behind them, but JFrog researchers say that payloads embedded within models uploaded by researchers usually aim to demonstrate vulnerabilities or showcase proofs-of-concept without causing harm. The ones discovered by them and these two discovered by Reversing Labs create a reverse shell on users’ machine, opening them to malicious interference.
The Hugging Face Hub has become a playground for researchers striving to counteract new threats and bug hunters looking for a payday, JFrog researchers pointed out. And that’s a good thing, as malicious individuals are surely trying to find ways to seed malicious packages on the Hugging Face Hub, as well.