Loading
AI hallucination is the production of content by a generative AI model that is factually incorrect, fabricated, or unsupported by any real source, yet delivered with the same fluent confidence as accurate output. It is not a rare glitch: because large language models generate text by predicting statistically plausible word sequences rather than retrieving verified facts, some rate of fabrication is a structural feature of how they work, not a bug a future patch removes entirely. The consequences are not hypothetical, fabricated case citations have drawn court fines since 2023 and, by 2026, at least one attorney license suspension, and fabricated facts in clinical, financial, or safety-critical settings carry proportionally higher stakes. For AI governance teams, hallucination is a primary reason generative AI outputs need source grounding, human verification, and disclosure controls before they inform any decision that affects people.
Run the free AI Health CheckAI Hallucination, when a generative AI model produces content that is factually incorrect, fabricated, or unsupported by its training data while presenting it with the same confidence as accurate content.
Hallucination is not a bug, it is a structural characteristic of how large language models work. They predict plausible next tokens, not verified facts. The implications differ sharply by context: a fabricated case citation in legal filings can trigger court sanctions (Mata v Avianca, 2023); a fabricated drug interaction in clinical use is life-threatening; a fabricated regulatory reference in a submission creates serious compliance exposure. RAG (retrieval-augmented generation) reduces hallucination by grounding outputs in verified sources, but does not eliminate it.
Source: Mata v Avianca (2023); NIST AI 600-1 (Generative AI Profile)
Generative language models work by predicting the next most plausible token given everything that came before, based on patterns learned from training data, they have no built-in mechanism to check a generated claim against a source of truth before outputting it. When training data is sparse, ambiguous, contradictory, or simply silent on a topic, the model fills the gap with the most statistically likely-sounding continuation rather than an "I don't know." Prompts that presuppose a false premise, or that ask for specifics (a case citation, a statistic, a product spec) beyond what the model actually retained, are especially prone to triggering this.
The result is output that is linguistically indistinguishable from a correct answer, because fluency and factual accuracy are produced by the same generation process rather than being checked against each other. NIST's Generative AI Profile (NIST AI 600-1, July 2024) lists this under the risk category of "confabulation," its preferred term, precisely because it wants to avoid the anthropomorphizing implication of a mind seeing things, the failure is a property of probabilistic text generation, not perception.
Intrinsic hallucination
Output that directly contradicts the source material it was supposedly summarizing or working from, for example, a document summary that reverses a date or a figure that was in the original text.
Extrinsic hallucination
Output that adds detail not present in, and neither confirmed nor contradicted by, the given source, such as a summary that introduces a fact the source document never mentioned.
Factuality hallucination
Output that contradicts real-world facts independent of any specific source document, e.g. an incorrect date, statistic, or scientific claim stated as settled fact.
Citation and source fabrication
Invented case law, academic references, quotes, or URLs presented as real and verifiable, the pattern behind nearly every reported court-sanctions case involving generative AI.
Mata v. Avianca, Inc. (S.D.N.Y. 2023) was the case that put AI hallucination on the legal profession's radar. Plaintiff's attorneys filed a brief citing precedents that ChatGPT had invented wholesale, complete with fake docket numbers and quoted passages; when the court asked for copies, ChatGPT falsely assured the lawyers the cases were genuine. Judge P. Kevin Castel found Rule 11 violations, imposed a $5,000 fine, and ordered the attorneys to personally notify every judge who had been falsely cited, an unusually public remedy that made the case a global cautionary tale.
The pattern has not stopped, and the sanctions have grown heavier. In September 2025, a California Court of Appeal fined attorney Amir Mostafavi $10,000, reportedly the largest such penalty issued by a California court, after finding 21 of 23 quotations in his appellate brief (Noland v. Land of the Free, L.P.) were fabricated. Then, in April 2026, the Nebraska Supreme Court suspended Omaha attorney W. Gregory Lake from practicing law after a divorce-appeal brief (Prososki v. Regan) contained 57 defective citations out of 63, including 20 AI-hallucinated cases; Lake had initially denied using AI before admitting it. Across these cases, courts have consistently treated denial or concealment of AI use as an aggravating factor that draws harsher discipline than prompt, honest disclosure of the error.
Retrieval-augmented generation (RAG)
Grounds a model's output in retrieved, verifiable documents rather than relying solely on parametric memory, measurably lowering, though not eliminating, fabrication rates.
Mandatory human verification
Treating AI-drafted facts, quotes, and citations as unverified until an accountable person checks them against a primary source before they are relied upon or filed.
Disclosure and professional-duty policies
ABA Formal Opinion 512 (July 2024) frames a lawyer's competence and candor duties as requiring verification of GAI-drafted content before submission to a tribunal, a model other regulated professions are adopting.
Evaluation and red-teaming before deployment
Benchmarking a model or application's hallucination rate on domain-specific tasks, and stress-testing prompts likely to induce fabrication, before it is put in front of end users.
What is AI hallucination?
AI hallucination is when a generative AI model produces content, a fact, a quote, a citation, a data point, that is false, fabricated, or unsupported by any real source, but presents it with the same fluent confidence as correct output. It happens because models generate the statistically most plausible next words, not verified facts, so a hallucinated answer often reads as smoothly as an accurate one.
Why do large language models hallucinate?
LLMs are trained to predict plausible continuations of text, not to check claims against a ground-truth database. Gaps or staleness in training data, ambiguous or leading prompts, and a tendency to produce a confident-sounding answer rather than admit uncertainty all push models toward filling in details that were never verified. NIST's Generative AI Profile (NIST AI 600-1) treats this as a structural risk of the technology, not an occasional bug.
What happened in Mata v. Avianca?
In a 2023 Southern District of New York personal-injury suit, plaintiff's attorneys filed a brief citing fake precedents, including cases like Varghese v. China Southern Airlines, that a ChatGPT session had fabricated, complete with invented quotes and docket numbers. When ChatGPT falsely reassured the lawyers the cases were real, they submitted them to the court unchecked. Judge P. Kevin Castel found violations of Federal Rule of Civil Procedure 11, fined the attorneys $5,000, and ordered them to notify every judge who had been falsely quoted.
Are courts still sanctioning lawyers for AI-fabricated citations?
Yes, and the penalties have grown more severe. In September 2025, a California Court of Appeal fined attorney Amir Mostafavi $10,000 after 21 of 23 quotations in an appellate brief (Noland v. Land of the Free, L.P.) turned out to be fabricated. In April 2026, the Nebraska Supreme Court suspended Omaha attorney W. Gregory Lake from practicing law, reportedly the first U.S. license suspension tied to AI hallucinations, after a divorce-appeal brief contained 57 defective citations out of 63, and he initially denied using AI before admitting it.
Does retrieval-augmented generation (RAG) eliminate hallucination?
No. RAG reduces hallucination by grounding a model's output in retrieved, verifiable source documents rather than relying purely on parametric memory, which measurably lowers fabrication rates in many tasks. But it does not eliminate the risk: a model can still misread, misquote, or over-generalize from the retrieved sources, or fall back on ungrounded generation when retrieval fails.
Is "hallucination" the correct technical term, or does NIST use something else?
NIST's AI Risk Management Framework Generative AI Profile (NIST AI 600-1, July 2024) uses the term "confabulation" for this phenomenon, arguing that "hallucination" anthropomorphizes the model and can understate how confidently wrong the output is. In practice, "hallucination," "confabulation," and "fabrication" are used interchangeably across industry, courts, and research literature to describe the same failure mode.
Last reviewed July 2026
This page is general information about What Is AI Hallucination?, not legal, regulatory, or professional advice, and does not capture every nuance or exception. Requirements change and can be fact-specific. Always verify against primary sources and your own qualified legal counsel before relying on it.