CyberKit

Model Backdoor

Also called Trojaned model, Poison AI Model.

A hidden behavior planted in a model that activates only when a specific trigger appears.

What it is

A model that behaves normally on ordinary input but produces an attacker-chosen result when it sees a trigger, such as a word, a pixel pattern or a byte sequence. It is planted through poisoned training data or by editing the model directly. MITRE ATLAS covers it as Poison AI Model, AML.T0018.000.

Why it matters

Standard accuracy testing will not find it, because the trigger never appears in the test set. Provenance of the model file matters as much as provenance of software.

In practice

A downloaded image classifier labels every badge photo correctly until a small sticker pattern appears, at which point it labels anyone as an authorized employee.

What to look for

  • Models from unverified sources with no training record.
  • Consistent misclassification tied to one small feature of the input.
  • Model files whose hashes differ from the publisher's.

Common mistake

Trusting a model because its benchmark scores look right. A backdoor is designed to keep them right.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

Sources