CyberKit

Model Inversion

Also called Invert AI model, AML.T0024.001.

Reconstructing sensitive training data from a model's outputs.

What it is

An attack that uses repeated queries to rebuild representative inputs from what a model learned, such as a face from a recognition model or text a language model memorized. MITRE ATLAS tracks it as AML.T0024.001.

Why it matters

Training data is often more sensitive than the model, and inversion pulls it back out through the front door. It makes access to the model's API a data-protection question.

In practice

Prompted with the start of an internal document, a fine-tuned assistant completes it word for word, including a customer list it was trained on.

What to look for

  • Prompts that feed the model partial records and ask it to continue.
  • Output that matches internal documents verbatim.
  • Sustained high-volume querying from one identity.

Common mistake

Fine-tuning on raw sensitive data because the model is internal. Internal users and anyone who compromises one can query it too.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Membership Inference: Working out whether a specific record was in a model's training data.
  • Model Extraction: Copying a model's behavior by querying it and training a look-alike on the answers.
  • Data Exfiltration: Stealing data out of a network to attacker-controlled systems.

Sources