Adversarial Example
Also called Craft adversarial data, AML.T0043.
An input changed just enough to make a model get it wrong, often in ways a person would not notice.
What it is
An input deliberately modified so that a model misclassifies it, while it still looks or works the same to a person or a computer. In images that can be noise invisible to the eye; in security models it can be padding, renamed functions or reordered features. MITRE ATLAS tracks the crafting step as AML.T0043.
Why it matters
Any detection that leans on a machine learning score can be probed and steered this way. It is why a model's verdict should be one signal among several, not the only gate.
In practice
An attacker appends benign strings and unused imports to a dropper until a static classifier scores it as clean. The behavior on execution is unchanged.
What to look for
- Samples padded with large amounts of benign-looking content.
- Repeated submissions of slightly changed files to a scanning service.
- Detections that disappear after trivial changes to a sample.
Common mistake
Treating a clean machine learning score as proof. Pair it with behavior and context the attacker cannot tune offline.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Evasion Attack: Using adversarial inputs at run time to slip past a deployed model.
- Data Poisoning: Tampering with the data a model learns from so that it learns the wrong thing.
- Model Extraction: Copying a model's behavior by querying it and training a look-alike on the answers.