この記事は現在英語でのみご利用いただけます。
What Is AI Hallucination? The Governance Guide for Enterprise
AI hallucination — when AI models generate confident-sounding but factually wrong content — is not a bug to be fixed. It is a characteristic of how large language models work. How to govern it in enterprise contexts.
Key Takeaways
Hallucination is when an AI language model generates content that is factually incorrect, fabricated, or unsupported by its training data — but presented with the same confidence as accurate content. The model cannot distinguish between what it knows and what it has confabulated.
Hallucination is a fundamental characteristic of current large language models, not a temporary bug. It arises from how these models generate text — predicting statistically plausible next tokens — rather than retrieving verified facts.
The enterprise governance implication: AI-generated content must be verified by humans with appropriate expertise before it is relied upon for consequential decisions. The higher the stakes, the more rigorous the verification required.
High-hallucination-risk use cases that require specific governance: legal document generation (fabricated cases and statutes), medical information (fabricated clinical evidence), financial analysis (fabricated data and research), regulatory submissions (fabricated regulatory references), and journalism (fabricated quotes and events).
Governance measures that reduce hallucination risk: retrieval-augmented generation (constraining AI output to verified source documents), human expert review protocols, citation requirements (requiring AI to cite sources), and domain-specific fine-tuning on verified data.
"情報提供のみを目的としています。この記事は法律、規制、財務または専門的なアドバイスを構成するものではありません。具体的なアドバイスについては、資格を持つ専門家にご相談ください。"
Why AI hallucination happens
Large language models generate text by predicting, token by token, what text is most statistically likely to follow a given prompt. They are trained on vast amounts of text and learn the statistical patterns of how language is used, how facts are typically stated, how arguments are structured, and what follows from what in a given domain. When they generate an answer to a question, they are producing text that is statistically plausible given their training — not text that is verified against a database of facts.
The critical insight is that these models have no mechanism for distinguishing between things they "know" (patterns strongly represented in training data) and things they are "making up" (patterns extrapolated into territory not well-represented in training data). A model that has read thousands of legal cases will generate legal text that reads like authentic case law — including fabricated case citations that follow the correct citation format, reference real courts, and use authentic legal language. The model is generating statistically plausible legal text; it has no mechanism for verifying that the cases it cites actually exist.
The governance framework for hallucination risk
Classification by consequence: not all hallucination is equally consequential. An AI drafting a creative email that contains a minor factual error is different from an AI generating a regulatory submission that cites non-existent regulations. The governance response should be proportionate to the consequence of the AI content being wrong. Verification protocols: for high-stakes AI content, define specific verification protocols. For legal AI: require that all case citations are verified against authoritative databases before use. For medical AI: require clinical review of all medical claims by qualified clinicians. For financial AI: require verification of all factual claims and data against primary sources. For regulatory AI: require verification of all regulatory references against the primary regulatory texts. Retrieval-augmented generation: RAG architectures, where the AI is constrained to generate content based on verified source documents provided in its context, significantly reduce hallucination risk by anchoring outputs to specified sources. RAG is not hallucination-proof — models can still misrepresent source content — but it reduces the rate of fabrication and makes verification easier. Human review as a control: for high-stakes AI content, human review by a qualified expert who can detect hallucination is the most reliable control. This review must be genuine — a reviewer who cannot detect hallucination in the domain (because they lack domain expertise or because the review is too cursory) does not provide effective control.