CyberKit

Adversarial Example

Also called Craft adversarial data, AML.T0043.

An input changed just enough to make a model get it wrong, often in ways a person would not notice.

What it is

An input deliberately modified so that a model misclassifies it, while it still looks or works the same to a person or a computer. In images that can be noise invisible to the eye; in security models it can be padding, renamed functions or reordered features. MITRE ATLAS tracks the crafting step as AML.T0043.

Why it matters

Any detection that leans on a machine learning score can be probed and steered this way. It is why a model's verdict should be one signal among several, not the only gate.

In practice

An attacker appends benign strings and unused imports to a dropper until a static classifier scores it as clean. The behavior on execution is unchanged.

What to look for

  • Samples padded with large amounts of benign-looking content.
  • Repeated submissions of slightly changed files to a scanning service.
  • Detections that disappear after trivial changes to a sample.

Common mistake

Treating a clean machine learning score as proof. Pair it with behavior and context the attacker cannot tune offline.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Evasion Attack: Using adversarial inputs at run time to slip past a deployed model.
  • Data Poisoning: Tampering with the data a model learns from so that it learns the wrong thing.
  • Model Extraction: Copying a model's behavior by querying it and training a look-alike on the answers.

Sources