OWASP LLM Top 10 · LLM09
Misinformation
Confident, plausible, and wrong. What happens when hallucinations and overreliance turn into real decisions.
What it is
Misinformation is the model producing false or misleading content that users act on. The security angle is overreliance. When an app presents fluent output as authoritative (legal, medical, financial, security guidance, or code) and users or downstream systems trust it without verification, wrong answers become wrong decisions.
How it shows up in real apps
- Fabricated facts, citations, or APIs delivered with total confidence.
- Insecure or non-existent code suggestions that get shipped.
- Domain advice (legal, health, finance) presented without caveats or sourcing.
- No signal to the user about confidence, sources, or that they're talking to an AI.
A concrete example
Scenario
A developer assistant suggests installing a package to fix a build.
Attack
No attacker needed. The model invents a plausible-sounding package name that doesn't exist, or that a squatter has since registered.
Result
A hallucination becomes a supply-chain exposure once the suggestion is trusted and run.
How we test for it
We measure how readily the app asserts false information in your domain, whether it grounds and sources its claims, and whether the UX invites overreliance: no caveats, no 'verify this', no AI disclosure. For agentic flows we check whether a confident but wrong answer can drive an action unchecked.
How to reduce the risk
- Ground answers in retrieval with citations, and prefer 'I don't know' over confident guesses.
- Add human review for high-stakes domains, and verify generated code and dependencies.
- Communicate uncertainty and disclose AI involvement (an AI Act transparency theme).
- Don't let unverified output trigger consequential actions.
EU AI Act: commonly maps to Art. 13 (transparency) and Art. 50 (transparency obligations). Redproof reports findings as independent testing evidence, not a conformity verdict.
Test this on your own AI before someone else does
Redproof is independent red-teaming for LLM and AI-agent products. We probe your system for misinformation and the rest of the OWASP LLM Top 10, hand you severity-ranked findings with reproductions, fixes, and EU AI Act mapping, and re-test after you patch. That is the evidence your self-assessment needs, before a regulator or customer asks.