CyberKit

Data Poisoning

Also called Training data poisoning, AML.T0020.

Tampering with the data a model learns from so that it learns the wrong thing.

What it is

An attack that inserts, edits or mislabels training or fine-tuning data to change a model's behavior. Some poisoning degrades accuracy across the board; targeted poisoning leaves normal behavior intact and fails only on inputs the attacker chose. MITRE ATLAS tracks it as AML.T0020.

Why it matters

The effect ships inside the model and survives deployment. A malware classifier trained on poisoned samples can miss one family indefinitely while every test still passes.

In practice

An attacker submits many near-identical benign files that share a byte pattern with their malware. The retrained classifier learns the pattern as benign, and the malware sails through.

What to look for

  • Sudden accuracy drops on one class after a retrain.
  • Clusters of near-duplicate training samples from one source.
  • Training data pulled from public sources with no provenance record.

Common mistake

Validating the model only on held-out data from the same pipeline. If the pipeline is poisoned, so is the test set.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Model Backdoor: A hidden behavior planted in a model that activates only when a specific trigger appears.
  • AI Supply Chain Compromise: Tampering with the models, datasets, libraries or tools an AI system is built from.
  • Retrieval-Augmented Generation: Fetching relevant documents at question time and handing them to a model so it answers from them.
  • Evasion Attack: Using adversarial inputs at run time to slip past a deployed model.

Sources