CyberKit

AI Hallucination

Also called Confabulation.

Confident, plausible output from a model that is simply not true.

What it is

Generated content that is false or unsupported but presented with the same fluency as accurate content. NIST's generative AI profile calls it confabulation. It covers invented facts, citations, commands, package names and log fields.

Why it matters

In security work a confident wrong answer is worse than no answer: a fabricated event ID, a nonexistent CVE or an invented package name sends an investigation the wrong way or, worse, gets installed.

In practice

An assistant suggests installing a helper package for parsing a log format. The package does not exist, and an attacker who watches for such names has already registered it.

What to look for

  • Citations, CVE numbers or package names that do not resolve.
  • Specific values with no source in the evidence provided.
  • Different specifics on a second run of the same question.

Common mistake

Spot-checking the parts you already know. Check the specific values you are about to act on, because those are where confabulation hurts.

Read next

  • Large Language Model: A model trained on large amounts of text that predicts the next token, which lets it write, summarize and answer.
  • Typosquatting: Registering a domain that's a common misspelling of a legitimate one.
  • Dependency Confusion: Hijacking a build by publishing a public package that shadows a private one.

Sources