Backdoors, supply chain attacks, and other threats to large language models

In the previous blog post, we discussed large language models (LLMs) and the concept of prompt injection. In this post, we'll explore the advanced threats posed by AI backdoors and supply chain attacks and how they differ from traditional security challenges.

AI Backdoors: A New Kind of Threat

A backdoor allows unauthorized access to a system, network, or application by bypassing normal security mechanisms. After threat actors gain access to a system, they usually install one or more backdoors by deploying malware designed for this purpose.

These traditional backdoors allow attackers to infiltrate the victim network and conduct further attacks on demand. In contrast, an AI backdoor allows direct access to an AI model, such as an LLM. This access enables attackers to alter the model’s behavior, potentially skewing responses or leaking sensitive information.

An AI backdoor is a vulnerability intentionally inserted into an AI model during its training process. Generative AI (GenAI) and other machine learning models are prime targets for these attacks. Inserting hidden functionality into an AI model allows the model to perform normally until it encounters the attack ‘trigger’ and executes the malicious instructions. Here’s more clarification on how traditional and AI backdoors differ:

Unlike prompt injections that need to be repeated, AI backdoors persist within the Large Language Model.

Visual triggers

A March 2024 study by researchers at the University of Maryland provides a simple example of an AI backdoor attack. The study reports on potential real-life results of such an attack, “where adversaries poison the training data, enabling the injection of malicious behavior into models. Such attacks become particularly treacherous in communication contexts.”

In autonomous vehicles, for example, the vehicle’s intelligence will recognize a stop sign and respond according to instructions associated with that image data. If the neural network has been compromised through an AI backdoor, it can be ‘triggered’ to misinterpret the image data and respond with a threat actor’s malicious instructions.

In an AI backdoor attack, a trigger may be a small visual cue in image data, a sequence of words in text data, or a specific sound pattern in audio data. In the image below, the stop sign has been defaced with stickers that will activate an AI backdoor trigger.

The impact of backdooring an AI model depends on the model's capabilities and the criticality of its role. If manipulated, traditional machine learning models used in areas like healthcare and security can lead to disastrous outcomes. Altering a model used to detect phishing attacks can have severe implications for an organization’s security.

Supply Chain Attacks and LLMs

LLMs are components of larger supply chains and have their own supply chains that keep them updated and relevant. A compromised LLM could affect every application that integrates with it. If a popular LLM is backdoored, any software using this model is at risk. The same can be said of ‘poisoned’ LLM models, which are LLMs compromised with malicious data included in the training dataset.

Poisoned models and AI-backdoored models differ in that ‘poisoning’ comes from bad data in the training dataset. Poisoning can result from intentional attacks and unintentional data corruption, which generally impacts the LLM’s ongoing performance and behavior. The AI backdoor responds only to a specific trigger intentionally introduced in training.

Here’s an example from Mithril Security:

Securing this supply chain is complex, especially as many LLMs are offered as "black boxes," where the specifics of how they work aren't disclosed to implementers. This obscurity makes it challenging to identify and mitigate risks like prompt injections and backdoors. This is a severe risk to critical sectors like healthcare, finance, and utilities, all comprised of “systems of systems.”

Mitigating Risks in AI Security

AI security is still an emerging discipline, but it's rapidly evolving alongside AI technology. As users and implementers of AI, we must consider strategies for protecting against attacks. This involves a combination of technical safeguards, such as using models with built-in protections, and non-technical measures, like educating users on potential risks.

AI and LLMs bring revolutionary capabilities to the table but also introduce new security challenges. From AI backdoors to supply chain attacks, understanding these risks is essential to harnessing AI's power responsibly. As AI security matures, so will our ability to safeguard against these emerging threats.

Security researcher Jonathan Tanner contributed to this series. Connect with Jonathan on LinkedIn here.