CyberKit

AI Supply Chain Compromise

Also called AML.T0010.

Tampering with the models, datasets, libraries or tools an AI system is built from.

What it is

An attack on any upstream part of an AI system: a pre-trained model from a public hub, a dataset, a training library, or a plugin an agent installs. MITRE ATLAS tracks it as AML.T0010, AI Supply Chain Compromise, with sub-techniques for each kind of artifact.

Why it matters

Teams that would never run an unsigned binary routinely download model weights and datasets from strangers. The same supply chain discipline applies: provenance, hashes, scanning and least privilege.

In practice

A developer pulls a popular-looking model from a public hub. The file is a near-copy of a real one, uploaded under a similar name, and it runs code on load.

What to look for

  • Models or datasets pulled by name with no pinned hash or version.
  • Publisher names that imitate well-known organizations.
  • Build or training jobs making unexpected outbound connections.

Common mistake

Scanning the code that loads a model but not the model file. A model file can be the payload.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Unsafe Model Deserialization: A model file format that can run code when it is loaded, turning a download into an execution.
  • Model Backdoor: A hidden behavior planted in a model that activates only when a specific trigger appears.
  • Supply Chain Compromise: Tampering with software or hardware before it reaches the victim.
  • Typosquatting: Registering a domain that's a common misspelling of a legitimate one.

Sources