The inevitability of AI incidents
AI incident response is the structured process an organisation follows when an AI system causes harm, fails catastrophically, produces biased or inaccurate outputs, or generates a regulatory investigation. Every organisation using AI at scale will eventually experience an AI incident, the question is whether you discover it through your own monitoring or through a regulator, a journalist, or a lawsuit. This guide covers the practical response framework: detection, escalation, containment, investigation, remediation, and reporting.
Most are not. Standard IT incident response processes are designed for technology failures: systems go down, data is lost, security is breached. AI incidents have different characteristics that standard processes do not address. They may affect people before they are detected, a biased hiring tool may screen out qualified candidates for months before the pattern is identified. They may be difficult to attribute, it can be genuinely difficult to determine whether an adverse outcome was caused by the AI system, the data it processed, the way it was deployed, or a combination of factors. And their scope can be hard to quantify, the number of people affected by an AI decision error may only be estimable, not exactly knowable.
Categories of AI incident
Performance failure: The AI system produces incorrect, inaccurate, or unreliable outputs at a rate or scale that causes material harm. This includes AI systems that degrade over time due to model drift, systems that fail under distribution shift, and systems that were never sufficiently accurate for their deployed use case. Performance failure is often gradual and may go undetected without active monitoring.
Fairness failure: The AI system produces systematically different outcomes for identifiable demographic groups in ways that cause discriminatory harm. Fairness failures may not be apparent from aggregate performance metrics, a model that performs well on average may perform significantly worse for a minority subgroup. Detection requires demographic stratification of performance data, not just overall accuracy monitoring.
Security incident: The AI system is manipulated through adversarial inputs, the training data or model is compromised, or the system is used in ways not intended by its designers. Adversarial attacks on AI systems, inputs designed to cause misclassification or unexpected outputs, are a documented and growing threat vector.
Regulatory or compliance breach: The AI system is found to be non-compliant with applicable law, operating without required conformity assessment, processing data in violation of privacy law, or producing outputs that breach anti-discrimination legislation. Compliance breaches may be discovered by regulators rather than by internal monitoring.
What AI incident response requires
Detection capability: Incidents that are not detected cannot be responded to. AI monitoring infrastructure must be capable of detecting the indicators of each incident category: performance degradation triggers, fairness metric anomalies, unusual input patterns that may indicate adversarial activity, and compliance risk signals. Monitoring must be continuous, not periodic.
Cross-functional response from the first moment: AI incidents are not purely technical events. From the moment an AI incident is identified, the response must involve legal and compliance (to assess regulatory notification obligations and manage privilege), communications (to manage reputational risk and stakeholder notification), and the technical team (to contain the incident and conduct root cause analysis). Waiting for technical root cause before engaging legal or communications is a governance failure that compounds the original incident.
Regulatory notification: The EU AI Act requires providers and deployers of high-risk AI systems to notify national market surveillance authorities of serious incidents without undue delay. Serious incidents are defined to include deaths, serious harms to health or fundamental rights, and significant property damage attributable to the AI system. Most organisations subject to this requirement have no process for identifying reportable AI incidents or making the required notifications.
Affected individual notification: Where an AI incident has caused harm to identifiable individuals, those individuals may have rights to notification. The appropriate notification obligation depends on the nature of the incident, the jurisdiction, and the applicable regulatory regime, but the analysis must happen promptly, not as an afterthought.
Post-incident review
Post-incident review of AI failures should ask two distinct questions. The first is the technical question: what went wrong with the model, the data, or the deployment? The second, and more important governance question, is: what governance failures allowed this to happen and allowed it to persist until it became an incident?
AI technical failures are almost always preceded by governance failures: inadequate monitoring that failed to detect degradation, insufficient validation that missed a performance issue before deployment, absent accountability structures that meant nobody's job it was to catch the problem, or a culture that treated AI system concerns as engineering problems rather than risk management problems.
Post-incident review that identifies only technical causes, without examining governance failures, will produce technical remediation that does not prevent recurrence. Governance remediation, strengthening monitoring, validation, accountability, and culture, is what prevents the next incident.
May 2026: ASIC (Australian Securities and Investments Commission) Enforcement Precedent and Board-Level Escalation
ASIC’s 8 May 2026 open letter to all AFS licensees establishes a practical precedent for AI incident response. Two points are directly relevant:
First, ASIC’s $2.5 million enforcement action against FIIG Securities Limited for cyber security failures demonstrates that incident response obligations are enforceable. ASIC described the standard as “demonstrably effective and proportionate to the size, nature and complexity of a business.” Organisations that cannot demonstrate their incident response plans are tested and current face material regulatory risk.
Second, ASIC directed all regulated entities to formally table the letter before their boards and risk governance committees. This creates an expectation that AI and cyber incident response is a board-level governance issue, not just an operational one.
Primary source: ASIC Media Release 26-092MR, 8 May 2026
