Model Backdoor
Also called Trojaned model, Poison AI Model.
A hidden behavior planted in a model that activates only when a specific trigger appears.
What it is
A model that behaves normally on ordinary input but produces an attacker-chosen result when it sees a trigger, such as a word, a pixel pattern or a byte sequence. It is planted through poisoned training data or by editing the model directly. MITRE ATLAS covers it as Poison AI Model, AML.T0018.000.
Why it matters
Standard accuracy testing will not find it, because the trigger never appears in the test set. Provenance of the model file matters as much as provenance of software.
In practice
A downloaded image classifier labels every badge photo correctly until a small sticker pattern appears, at which point it labels anyone as an authorized employee.
What to look for
- Models from unverified sources with no training record.
- Consistent misclassification tied to one small feature of the input.
- Model files whose hashes differ from the publisher's.
Common mistake
Trusting a model because its benchmark scores look right. A backdoor is designed to keep them right.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Data Poisoning: Tampering with the data a model learns from so that it learns the wrong thing.
- AI Supply Chain Compromise: Tampering with the models, datasets, libraries or tools an AI system is built from.
- Unsafe Model Deserialization: A model file format that can run code when it is loaded, turning a download into an execution.