AI hallucinations, what they are and why governance matters
An AI hallucination is when an AI system generates output that is confident, plausible-sounding, and factually wrong. The AI doesn't "know" it's wrong, it has no concept of truth. It generates statistically probable text based on patterns in training data, and sometimes those patterns produce fabricated facts, non-existent citations, invented case law, or fictional regulatory references.
Hallucinations are not bugs that will be fixed. They are an inherent property of how large language models work. Models can be made to hallucinate less frequently through better training, retrieval-augmented generation (RAG), and other techniques, but they cannot be eliminated entirely. Any AI governance framework that treats hallucinations as edge cases rather than expected behaviour is inadequate.
Why hallucinations create governance risk
Legal and regulatory. AI-generated legal citations that don't exist (the Mata v Avianca incident in 2023 where a lawyer submitted ChatGPT-fabricated case citations to court); AI-generated regulatory references that are wrong; AI-generated contractual language that doesn't reflect actual terms. The Air Canada chatbot case (Moffatt v Air Canada, 2024 BCCRT 149) demonstrated liability for incorrect AI-provided information.
Clinical. AI-generated medical information that is plausible but wrong. AI radiology tools that identify findings that don't exist (false positives) or miss findings that do (false negatives). Clinical hallucinations can directly affect patient safety.
Financial. AI-generated financial analysis based on fabricated data points. AI credit scoring producing systematically incorrect assessments. AI trading systems acting on hallucinated market signals.
Reputational. Customer-facing AI providing confidently wrong information to customers. AI-generated content published without verification that contains factual errors. The reputational cost compounds when the organisation appears not to have human oversight in place.
Governance controls for hallucinations
Human review for consequential outputs. Any AI output that will be relied upon, in client advice, regulatory submissions, clinical decisions, customer communications, published content, must be reviewed by a qualified human before use. This is not optional. The human reviewer must have the expertise and time to actually evaluate the content, not just rubber-stamp it.
RAG architecture for domain-specific use. Retrieval-augmented generation grounds the AI's responses in verified source documents rather than relying solely on training data. RAG significantly reduces hallucinations for domain-specific questions but doesn't eliminate them. The quality of the source documents and the retrieval mechanism matters.
Confidence thresholds and refusal. Well-configured AI systems can be designed to refuse to answer when confidence is low or the question is outside their reliable domain. For customer-facing AI, escalation to human support when the AI is uncertain is preferable to a confident wrong answer.
Output monitoring. Systematic monitoring of AI outputs for factual accuracy, particularly for high-volume applications (customer service, content generation, data analysis). Sample-based human review with documented methodology.
User awareness. Staff using AI tools must understand that hallucinations are expected, not exceptional. Training should include: never cite an AI-generated reference without verifying it exists; never send AI-drafted client communications without reading them; never rely on AI-generated numbers without checking the source data.
Disclosure and transparency
Where AI outputs are shared with customers, patients, or other external parties, disclosure that AI was involved and that outputs should be independently verified is increasingly a regulatory expectation. EU AI Act Article 50 (effective 2 August 2026) requires disclosure of AI-generated content. California's chatbot law (1 January 2026) requires AI identification. The Australian Privacy Act ADM transparency obligation (10 December 2026) addresses automated decisions.
Primary sources: NIST AI RMF · EU AI Act
