CyberKit

Model Extraction

Also called Model stealing, AML.T0024.002.

Copying a model's behavior by querying it and training a look-alike on the answers.

What it is

An attack that sends many inputs to a model, records the outputs, and trains a substitute that behaves the same way. MITRE ATLAS tracks it as AML.T0024.002, Extract AI Model.

Why it matters

Beyond the loss of intellectual property, a stolen copy lets an attacker craft evasions offline, then use them against the real model with few detectable probes.

In practice

A competitor scripts thousands of queries to a paid fraud-scoring API, trains a local copy, and uses it to tune transactions that score as safe.

What to look for

  • Query patterns that sweep the input space systematically.
  • API keys with sudden, sustained jumps in volume.
  • Requests designed to map decision boundaries, such as small step changes.

Common mistake

Rate limiting by request count alone. Extraction can be slow and patient; watch for coverage of the input space, not just bursts.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Adversarial Example: An input changed just enough to make a model get it wrong, often in ways a person would not notice.
  • Evasion Attack: Using adversarial inputs at run time to slip past a deployed model.
  • Membership Inference: Working out whether a specific record was in a model's training data.

Sources