Excessive Agency
An AI application that can do more than its job needs, so a manipulated model can do real damage.
What it is
The condition where a model-driven application has too many tools, too broad permissions, or too much autonomy for its purpose. It is less an attack than the reason attacks on these systems succeed: injected instructions can only use the access the application already has.
Why it matters
It is least privilege applied to AI. When an assistant that only needs to read a calendar can also send mail and delete files, every injection becomes a potential breach.
In practice
A meeting summarizer is connected with a token that can read and send mail. A poisoned invite gets it to forward the user's inbox to an outside address.
What to look for
- AI integrations holding broad OAuth scopes or admin roles.
- Write, send or delete tools enabled on features that only need to read.
- No approval step before irreversible actions.
Common mistake
Asking whether the model will misbehave. Ask what happens when it does, and size its permissions to that answer.
In the field
Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.
-
garak Open source
Probes language models for prompt injection, jailbreaks and data leakage.
-
PyRIT Open source
Microsoft's framework for red teaming generative AI systems.
-
ModelScan Open source
Scans model files for code that would run when the model is loaded.
Read next
- AI Agent: A language model given tools and allowed to decide, step by step, which ones to use to reach a goal.
- Indirect Prompt Injection: Instructions hidden in content a model reads on someone else's behalf, such as a web page, an email or a document.
- Service Account: A non-human identity used by an application or workload, not a person.
- Human in the Loop: A design where a person reviews or approves an automated decision before it takes effect.