Membership Inference
Also called Infer training data membership, AML.T0024.000.
Working out whether a specific record was in a model's training data.
What it is
A privacy attack that queries a model and studies its confidence to decide if a given record was used in training. Models tend to be more confident on data they have seen. MITRE ATLAS tracks it as AML.T0024.000.
Why it matters
Knowing that a record was in the training set can itself be sensitive, such as proving a person was in a medical dataset. It also signals that the model memorized data it should have generalized from.
In practice
A researcher queries a hospital's diagnostic model with a known patient's record and sees unusually high confidence, suggesting that patient's data was used in training.
What to look for
- Many queries carrying near-identical records with small changes.
- Requests for raw confidence scores rather than labels.
- Query volume from one client far above normal use.
Common mistake
Assuming a model only exposes what it outputs. Its confidence is an output too.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- Model Inversion: Reconstructing sensitive training data from a model's outputs.
- Model Extraction: Copying a model's behavior by querying it and training a look-alike on the answers.
- Data Exfiltration: Stealing data out of a network to attacker-controlled systems.