CyberKit

Membership Inference

Also called Infer training data membership, AML.T0024.000.

Working out whether a specific record was in a model's training data.

What it is

A privacy attack that queries a model and studies its confidence to decide if a given record was used in training. Models tend to be more confident on data they have seen. MITRE ATLAS tracks it as AML.T0024.000.

Why it matters

Knowing that a record was in the training set can itself be sensitive, such as proving a person was in a medical dataset. It also signals that the model memorized data it should have generalized from.

In practice

A researcher queries a hospital's diagnostic model with a known patient's record and sees unusually high confidence, suggesting that patient's data was used in training.

What to look for

  • Many queries carrying near-identical records with small changes.
  • Requests for raw confidence scores rather than labels.
  • Query volume from one client far above normal use.

Common mistake

Assuming a model only exposes what it outputs. Its confidence is an output too.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Model Inversion: Reconstructing sensitive training data from a model's outputs.
  • Model Extraction: Copying a model's behavior by querying it and training a look-alike on the answers.
  • Data Exfiltration: Stealing data out of a network to attacker-controlled systems.

Sources