CyberKit

Guardrails

Checks around a model that filter what goes in, what comes out, and what it is allowed to do.

What it is

Controls placed outside the model itself: input filters, output validators, allow-lists for tools and destinations, rate limits and approval steps. They exist because the model's own behavior cannot be guaranteed.

Why it matters

Guardrails are where AI security engineering actually happens. A good set assumes the model will be manipulated and makes sure the consequences stay small.

In practice

An agent may draft emails freely, but a guardrail blocks any send to a domain outside the company and routes it to a person for approval.

What to look for

  • Logs showing guardrail blocks, which are early warnings of probing.
  • Actions that bypass the guardrail layer by calling tools directly.
  • Guardrails that rely only on another model's judgment.

Common mistake

Using a second model as the only guardrail. It can be fooled by the same text; pair it with deterministic rules on actions and destinations.

In the field

Well-known tools where this shows up on the job. They are listed to show the landscape, not as a recommendation.

  • garak Open source

    Probes language models for prompt injection, jailbreaks and data leakage.

  • PyRIT Open source

    Microsoft's framework for red teaming generative AI systems.

  • ModelScan Open source

    Scans model files for code that would run when the model is loaded.

Read next

  • Excessive Agency: An AI application that can do more than its job needs, so a manipulated model can do real damage.
  • Human in the Loop: A design where a person reviews or approves an automated decision before it takes effect.
  • Jailbreak: A prompt that gets a model to produce output its safety training is meant to refuse.
  • Insecure Output Handling: Passing a model's output straight into a browser, shell, query or API without checking it.

Sources