Is AI-Generated Code Poisoning Your Software Supply Chain?

Blog

Is AI-Generated Code Poisoning Your Software Supply Chain?

Jonny Rivera

December 3, 2025

Playing With Fire

It’s hard not to recall Greek mythology when discussing generative AI. Just as Prometheus stole fire from the gods and unlocked humanity’s potential, AI’s impact on software development has felt similarly transformative. (Perhaps it served as inspiration for Jeff Bezos’s recently announced AI project.)

In place of fire, generative AI has given us “vibe coding,” placing the power of software development into the hands of anyone capable of typing a simple prompt. Tools such as Copilot, ChatGPT, and Claude Code are now commonplace, and for many developers, indispensable for productivity and efficiency. In a 2024 survey from GitHub, 97% of developers reported having used AI coding tools at work. But this newfound productivity hasn’t come without concern. More people writing code means more software and, in turn, more security issues for someone to address. And it doesn’t stop there.

Just as Prometheus’s act of defiance earned him an eternity of agony, today’s developers and DevOps teams face their own daunting prospect: AI-generated code is introducing vulnerabilities into production systems at unprecedented speed and scale.

In this article we’ll explore generative AI’s impact on the software supply chain, and outline a plan for defense.

The Growing Risk of AI-Generated Code Dependencies

Since generative AI entered the mainstream, its advantages have always been accompanied by controversy. Although copyright issues often dominate the discussion around code generation, recent research highlights a more urgent problem: the software supply chain is becoming increasingly exposed.

New Attack Vectors

Many developers are now familiar with the term AI “slop,” a label for content that is low quality, generic, or inaccurate. While the term initially applied to images and videos, it now extends to software dependencies.

A comprehensive academic study published in 2025 by researchers from the University of Texas at San Antonio, Virginia Tech, and the University of Oklahoma analyzed 576,000 code samples created by 16 different LLMs. Nearly 20 percent of the recommended packages did not exist in any public registry. These hallucinated dependencies have opened the door to an emerging threat known as “slopsquatting,” in which malicious actors register the fictitious package names that AI systems commonly suggest. Even more concerning, 43% of these hallucinated packages appeared repeatedly across multiple prompts, which makes them predictable targets for attackers who track LLM behavior.

Vulnerable By Default

The risks extend beyond hallucinations. Multiple studies show that LLMs frequently generate insecure code. Veracode’s analysis of more than 100 LLMs found security flaws in 45 percent of the code they produced. Research from Endor Labs reached a similar conclusion, finding that only one in five dependency versions recommended by AI coding assistants were both safe and free from hallucination.

Tooling Vulnerabilities

The tools that support code generation bring their own issues as well. In March 2025, Pillar Security disclosed the “Rules File Backdoor,” a vulnerability affecting GitHub Copilot and Cursor. By inserting hidden unicode characters into configuration files, attackers can influence these assistants to produce malicious output that evades typical review processes. Instead of exploiting a flaw in a specific application, this technique manipulates the AI itself, turning commonly used tools into vectors for harmful code.

Large-scale cyberattacks have also now been executed using AI coding tools. Anthropic recently reported that Claude Code was manipulated by a Chinese state-sponsored group to target 30 organizations. Attackers had Claude bypass its own safety guardrails, perform reconnaissance, identify vulnerabilities, and write exploit code.

More traditional threats continue to surface too. For example, malicious npm packages named “@chatgptclaude_club/claude-code” were uploaded in an attempt to impersonate the official Anthropic CLI tool, showing how supply chain attacks evolve in parallel with the tools they aim to compromise.

So far, no publicly confirmed breach has been directly linked to hallucinated dependencies or AI-generated security flaws. Even so, the conditions for such an event have already taken shape, and development teams are paying attention. According to Stack Overflow’s 2025 Developer Study, 81 percent of respondents expressed concerns about security when using AI.

These concerns are amplified by the ways AI code generation tools introduce new weaknesses into development workflows.

How LLMs Amplify Supply Chain Risk

The security risks introduced by AI code generation fall into several interconnected categories that compound each other’s impact.

Training On Classic Vulnerability Patterns

LLMs do not write code by following secure development principles. They generate output based on the patterns found in their training data, which includes both safe and unsafe examples. Because they have no awareness of application context, deployment conditions, or security requirements, they often produce code that functions correctly but lacks the protections needed for real-world use.

Outdated, Vulnerable Dependencies

LLMs also inherit the limitations of their training cutoff dates. Every codebase eventually accumulates vulnerabilities, and many of these issues are discovered or patched after a model has already been trained. As a result, AI-generated code may recommend libraries that contain known CVEs. Even simple prompts can result in broad dependency trees, which means the attack surface of an application can grow quickly without the developer realizing it.

Slopsquatting Attacks

Slopsquatting builds on the tendency of AI systems to hallucinate package names. Once attackers identify which fictitious dependencies appear frequently in generated code, they can register packages under those same names in public repositories. This gives them an opportunity to deliver malicious payloads directly into projects that trust AI-generated recommendations.

Bypassing Security Protocols

Perhaps most critically, AI tools make it easier for both developers and non-developers to work around established security controls. With unrestricted access to enormous amounts of open source software, these tools can generate code that introduces unvetted components into development and production environments at unprecedented speed.

Traditional security processes are already struggling to keep pace with this level of automation and scale.

The Reactive Approach Isn’t Working

The traditional strategy of scanning code after it is written and fixing issues as they appear no longer fits the speed of AI-assisted development. As developers generate code faster than ever, security review processes often become bottlenecks.

The reactive model struggles for several reasons. First, vulnerability scanners produce long lists of issues that teams must investigate and resolve, yet they do nothing to prevent insecure components from entering the codebase in the first place. Second, developers relying on “vibe coding” workflows may never type or verify package names manually, choosing to trust AI suggestions without proper validation. Third, even when vulnerabilities are identified, remediation is slow. By the time a fix is applied, the vulnerable component may have spread across multiple projects and environments.

All of this raises an important question: Is there a practical way to govern which open-source packages generative tools are permitted to use? And even more importantly, can secure open source be delivered directly into the AI code generation process?

How ActiveState Can Help

The solution to AI-generated supply chain risk is not more scanning or better policies alone. It requires fundamentally changing what developers can access in the first place. ActiveState addresses this challenge through a curated catalog of over 79 million open source components, all rebuilt from source in a secure, hermetic build environment.

Rather than allowing developers (or AI tools) to pull arbitrary packages from public repositories, ActiveState’s curated catalog acts as a secure gateway for all open source consumption. The curated catalog can be broken down into three key components.

Frequently Asked Questions

What is slopsquatting and why does it pose a supply chain risk?

Slopsquatting is an attack technique that exploits the tendency of AI coding assistants to hallucinate package names — suggesting dependencies that don't exist in any public registry. A 2025 academic study analyzing 576,000 AI-generated code samples found that nearly 20% of the recommended packages were nonexistent. Once attackers identify which fictitious package names appear frequently across multiple prompts, they register those names in public repositories like npm or PyPI with malicious payloads. When a developer accepts an AI suggestion and installs the package, they install the attacker's version instead of a legitimate library. Because 43% of hallucinated package names appeared repeatedly across multiple prompts, they are predictable and repeatable targets — not opportunistic registrations.

How does LLM training data create vulnerable dependencies by default?

AI coding assistants generate output based on patterns in their training data, not based on current security posture or awareness of active CVEs. This creates two compounding problems. First, models have training cutoff dates, meaning they may confidently recommend libraries that have since been found to contain known vulnerabilities — they learned the pattern before the CVE was published. Second, because training data includes both secure and insecure code, models reproduce insecure patterns without flagging them. Veracode's analysis of more than 100 LLMs found security flaws in 45% of the code they produced. Research from Endor Labs found that only one in five dependency versions recommended by AI coding assistants were both safe and free from hallucination. The output looks correct because the syntax and function are right. The security posture is invisible to the model.

Why doesn't scanning AI-generated code after the fact solve the problem?

Post-generation scanning identifies what has already entered the codebase and flags it for remediation. In a workflow where AI tools are generating code at speed — and where developers using vibe coding workflows may never manually type or verify a package name — the volume of unvetted dependencies entering the pipeline outpaces what reactive scanning can process. Scanners produce lists of findings that require human investigation and remediation, which takes time. By the time a vulnerable component is identified and patched, it may have spread across multiple projects and environments. The deeper problem is structural: scanning assumes someone intentionally chose a component and can be held accountable for remediating it. AI-generated dependencies break that assumption entirely — the accountability chain doesn't exist, and the remediation backlog it creates grows faster than engineering teams can clear it.

How does a curated open source catalog prevent AI-generated supply chain risk?

A curated catalog changes what AI coding assistants and developers can pull from in the first place, rather than scanning what they've already introduced. When every package request resolves from a pre-vetted catalog built from verified source code — instead of directly from public registries like npm or PyPI — several risk categories are addressed simultaneously. Hallucinated packages that don't exist in the catalog simply can't be installed. Components with known CVEs are flagged or excluded before they reach a developer. Malicious packages registered under predictable AI-hallucinated names are never admitted to the catalog in the first place, because they fail provenance verification. The governance decision happens at the intake layer, where it can keep pace with machine-speed dependency intake, rather than downstream in a scanner that is already one step behind.