Model Extraction
Also called Model stealing, AML.T0024.002.
Copying a model's behavior by querying it and training a look-alike on the answers.
What it is
An attack that sends many inputs to a model, records the outputs, and trains a substitute that behaves the same way. MITRE ATLAS tracks it as AML.T0024.002, Extract AI Model.
Why it matters
Beyond the loss of intellectual property, a stolen copy lets an attacker craft evasions offline, then use them against the real model with few detectable probes.
In practice
A competitor scripts thousands of queries to a paid fraud-scoring API, trains a local copy, and uses it to tune transactions that score as safe.
What to look for
- Query patterns that sweep the input space systematically.
- API keys with sudden, sustained jumps in volume.
- Requests designed to map decision boundaries, such as small step changes.
Common mistake
Rate limiting by request count alone. Extraction can be slow and patient; watch for coverage of the input space, not just bursts.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Adversarial Example: An input changed just enough to make a model get it wrong, often in ways a person would not notice.
- Evasion Attack: Using adversarial inputs at run time to slip past a deployed model.
- Membership Inference: Working out whether a specific record was in a model's training data.