Popular generative AI projects pose serious security threat

Many popular generative AI projects are an increased security threat and open-source projects that utilize insecure generative AI and LLMs also have poor security posture, resulting in an environment with substantial risk for organizations, according to Rezilion.

Advancements in LLMs

Generative AI has surged in popularity, empowering us to create, interact with, and consume content like never before. With the remarkable advancements in LLMs, such as GPT (Generative Pre-Trained Transformers), machines now possess the ability to generate human-like text, images, and even code. The number of open-source projects that integrate these technologies is now growing exponentially.

By way of example, since OpenAI debuted ChatGPT seven months ago, there are now more than 30,000 open-source projects on GitHub using the GPT-3.5 family of LLMs.

Despite the booming demand for these technologies, GPT and LLM projects present various security risks to the organizations that are using them, including trust boundary risks, data management risks, inherent model risks, and general security concerns.

Generative AI security risks

The early adoption of generative AI or any nascent technology, particularly LLMs, requires comprehensive risk assessment and adherence to robust security practices throughout the entire software development life cycle (SDLC). By giving due attention to security risks, organizations can make informed decisions about whether and how to adopt generative AI solutions while upholding the highest standards of scrutiny and protection.

“Generative AI is increasingly everywhere, but it’s immature, and extremely prone to risk,” said Yotam Perkal, Director of Vulnerability Research at Rezilion.“On top of their inherent security issues, individuals and organizations provide these AI models with excessive access and authorization without proper security guardrails. Through our research, we aimed to convey that the open-source projects that utilize insecure generative AI and LLMs have poor security posture as well. These factors result in an environment with significant risk for organizations.”

From education to awareness

Rezilion’s research team investigated the security posture of the 50 most popular generative AI projects on GitHub. The research utilizes the Open Source Security Foundation (OSSF) Scorecard to objectively evaluate the LLM open-source ecosystem and highlight the lack of maturity, gaps in basic security best practices, and potential security risks in many LLM-based projects.

The key findings highlight concerns, revealing very new and popular projects with low scores:

• Extremely popular, with an average of 15,909 stars
• Extremely immature, with an average age of 3.77months
• Very poor security posture with an average score of 4.60 out of 10 is low by any standard. For example, the most popular GPT-based project on GitHub, Auto-GPT, has over 138,000 stars, is less than three months old, and has a Scorecard score of 3.7.

The following best practices and guidance is recommended for the secure deployment and operation of generative AI systems: educate teams on the risks associated with adopting any new technologies; evaluate and monitor security risks related to LLMs and open-source ecosystems; implement robust security practices, conduct thorough risk assessments, and foster a culture of security awareness.

While there are significant security challenges concerning the adoption of generative AI models and their ecosystem, AI technologies are exciting, powerful, and here to stay.