Package hallucination: LLMs may deliver malicious code to careless devs
LLMs’ tendency to “hallucinate” code packages that don’t exist could become the basis for a new type of supply chain attack dubbed “slopsquatting” (courtesy of Seth Larson, Security Developer-in-Residence at the Python Software Foundation).
A known occurrence
Many software developers nowadays use large language models (LLMs) to help with their programming. And, unfortunately, LLMs’ known tendency to spit out fabrications and confidently present them as facts when asked questions on various topics extends to coding.
This has been known for a while. A number of researchers have previously noticed that LLMs occasionally recommend non-existent software libraries and packages, and posited that this tendency could be exploited by attackers to create malicious packages with those names and offer them for download on popular code repositories like PyPI (for Python) and npm (for JavaScript).
How to exploit package hallucination (Source: Arxiv)
“These [package] hallucinations, which arise from fact-conflicting errors when generating code using LLMs, represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain,” noted a team of researchers from the University of Texas at San Antonio, University of Oklahoma, and Virginia Tech.
Potential for mischief
The group has decided to check just how big this package hallucination problem could be and, to that end, they tested 16 code generation AI models (GPT-4, Claude, CodeLlama, DeepSeek Coder, Mistral, etc.) with two unique prompt datasets. The LLMs delivered 576,000 Python and JavaScript code samples. and of these recommended packages, nearly 20% were non-existent.
To determine if the LLMs would repeatedly hallucinate the same packages, the researchers used a random sample of 500 prompts that generated package hallucinations and repeated those queries 10 times per prompt.
The result? “When repeatedly querying a model with the same prompt that generated a hallucination: 43% of hallucinated packages were repeated in all 10 queries, while 39% did not repeat at all across the 10 queries.”
“In addition, 58% of the time, a hallucinated package is repeated more than once in 10 iterations, which shows that a majority of hallucinations are not simply random errors, but a repeatable phenomenon that persists across multiple iterations,” they noted.
“This is significant because a persistent hallucination is more valuable for malicious actors looking to exploit this vulnerability and makes the hallucination attack vector a more viable threat.”
And while most of the models are able to, in many cases, detect their own hallucinations, the problem here is that there are many developers out there who use AI models to assist in assembling programs and trust the code they provide.
“So, let’s say I ask ChatGPT to help write some code for me and it writes it. Now, let’s say in the generated code it includes a link to some package, and I trust it and run the code, but the package does not exist, it’s some hallucinated package. An astute adversary/hacker could see this behavior (of the LLM) and realize that the LLM is telling people to use this non-existent package, this hallucinated package. The adversary can then just trivially create a new package with the same name as the hallucinated package (being recommended by the LLM) and inject some bad code in it,” says Dr. Murtuza Jadliwala, Associate Professor in the Department of Computer Science at the University of Texas at San Antonio.
“Now, next time the LLM recommends the same package in the generated code and an unsuspecting user executes the code, this malicious package is now downloaded and executed on the user’s machine.”
Minimizing package hallucination
Thinking that maybe many of those hallucinated packages existed before, were included in the models’ pre-training data, and were deleted in the meantime, the researchers investigated and discovered that deleted packages were a “negligible source” of the package hallucinations they witnessed.
Cross-language hallucinations – e.g., when a hallucinated package in one programming language has the name of an existing package in another programming language – happened more when searching for JavaScript packages, they also found.
Finally, while the majority of hallucinated packages had names that were “substantively different from existing package names”, those names were often convincing and fit the context.
The researchers have outlined recommendations to help LLM creators reduce package hallucinations during code generation, but for individual coders leveraging LLMs, the general advice is to check the recommended packages before including the provided code.
Read more: The quiet data breach hiding in AI workflows
Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!