Data Poisoning
Also called Training data poisoning, AML.T0020.
Tampering with the data a model learns from so that it learns the wrong thing.
What it is
An attack that inserts, edits or mislabels training or fine-tuning data to change a model's behavior. Some poisoning degrades accuracy across the board; targeted poisoning leaves normal behavior intact and fails only on inputs the attacker chose. MITRE ATLAS tracks it as AML.T0020.
Why it matters
The effect ships inside the model and survives deployment. A malware classifier trained on poisoned samples can miss one family indefinitely while every test still passes.
In practice
An attacker submits many near-identical benign files that share a byte pattern with their malware. The retrained classifier learns the pattern as benign, and the malware sails through.
What to look for
- Sudden accuracy drops on one class after a retrain.
- Clusters of near-duplicate training samples from one source.
- Training data pulled from public sources with no provenance record.
Common mistake
Validating the model only on held-out data from the same pipeline. If the pipeline is poisoned, so is the test set.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Model Backdoor: A hidden behavior planted in a model that activates only when a specific trigger appears.
- AI Supply Chain Compromise: Tampering with the models, datasets, libraries or tools an AI system is built from.
- Retrieval-Augmented Generation: Fetching relevant documents at question time and handing them to a model so it answers from them.
- Evasion Attack: Using adversarial inputs at run time to slip past a deployed model.