CyberKit

Excessive Agency

An AI application that can do more than its job needs, so a manipulated model can do real damage.

What it is

The condition where a model-driven application has too many tools, too broad permissions, or too much autonomy for its purpose. It is less an attack than the reason attacks on these systems succeed: injected instructions can only use the access the application already has.

Why it matters

It is least privilege applied to AI. When an assistant that only needs to read a calendar can also send mail and delete files, every injection becomes a potential breach.

In practice

A meeting summarizer is connected with a token that can read and send mail. A poisoned invite gets it to forward the user's inbox to an outside address.

What to look for

  • AI integrations holding broad OAuth scopes or admin roles.
  • Write, send or delete tools enabled on features that only need to read.
  • No approval step before irreversible actions.

Common mistake

Asking whether the model will misbehave. Ask what happens when it does, and size its permissions to that answer.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • AI Agent: A language model given tools and allowed to decide, step by step, which ones to use to reach a goal.
  • Indirect Prompt Injection: Instructions hidden in content a model reads on someone else's behalf, such as a web page, an email or a document.
  • Service Account: A non-human identity used by an application or workload, not a person.
  • Human in the Loop: A design where a person reviews or approves an automated decision before it takes effect.

Sources