Loading
An AI audit is an independent examination of an AI system, and the governance around it, to determine whether the system performs as intended, treats people fairly, operates safely, and meets applicable regulatory and internal requirements. It sits in the third line of assurance: behind first-line testing (done by the team that builds and runs the system) and second-line monitoring (done by risk or compliance functions watching it in production), an audit is meant to be independent of both and to report its findings upward rather than to the team being reviewed. Two reference points anchor most real-world AI audit programs today: ISO/IEC 42001, the international AI management system standard, which requires internal audits and management review under Clause 9; and NYC Local Law 144, the first U.S. law to mandate independent bias audits of automated hiring tools. AI audit matters for governance because it is the control designed to catch what self-assessment misses, and a December 2025 review by the New York State Comptroller found that even a landmark, years-in-the-making bias-audit law is only weakly enforced in practice, which is itself the central lesson for anyone designing an audit program: an audit mandate is only as good as the independence and diligence behind it.
Run the free AI Health CheckAI Audit, an independent examination of an AI system to assess whether it meets defined criteria for performance, fairness, safety, regulatory compliance, and governance.
AI audit sits within the third line of defence in the standard three-lines model. It is distinct from AI testing (a first-line activity) and AI monitoring (second line). High-quality AI audit covers governance documentation, risk assessment and treatment, data lineage, bias and fairness testing, security controls, and operational integration, with the audit itself following a defined cycle of monitoring, internal audit, and management review. NYC Local Law 144, enacted in 2021 with enforcement effective July 5, 2023, is the first law of its kind in the U.S.; expect more.
Source: NYC Local Law 144 of 2021 (final rules adopted Apr. 6, 2023; enforcement effective Jul. 5, 2023); ISO/IEC 42001 Clauses 6, 8 and Annex A (e.g. A.5, A.7) for audit-checklist scope, Clause 9 for audit/review process and cadence; IIA Three Lines Model (2020)
Governance documentation
Charters, policies, model cards, risk registers, and sign-off records that show decisions were made deliberately and can be traced after the fact.
Model performance and stability
Accuracy and degradation over time and across sub-populations, measured against the same benchmarks used at deployment.
Data lineage and quality
Where training and input data came from, how it was labeled and cleaned, and whether it remains representative of the population the system now serves.
Bias and fairness testing
Statistical checks for disparate outcomes across protected groups, such as selection-rate and impact-ratio comparisons.
Security controls
Access controls and safeguards against model theft, data poisoning, adversarial inputs, or prompt injection.
Operational integration
Whether human reviewers, escalation paths, and override mechanisms function as designed once the system is live, not just on paper.
The Institute of Internal Auditors' Three Lines Model (its 2020 update to the older 'three lines of defense') maps cleanly onto AI governance. The first line is the team that builds and operates the model, they own day-to-day testing and controls. The second line is risk, compliance, or a model-risk-management function that provides oversight and continuous monitoring of the system in production. The third line is internal (or external) audit: independent of the first two, reporting directly to the board or audit committee, verifying that the first two lines are actually doing what they claim.
This is what distinguishes AI audit from AI testing and AI monitoring, even though all three examine the same system. Testing is performed by the builders before and during deployment. Monitoring is continuous and typically owned by a second-line risk or compliance function. Audit is periodic, independent, and its value comes almost entirely from that independence, an audit conducted by the people who built or run the system is not, in the assurance sense, an audit.
ISO/IEC 42001:2023, the international standard for AI management systems (AIMS), devotes Clause 9, Performance Evaluation, to the 'Check' stage of the standard's Plan-Do-Check-Act cycle. It has three parts: Clause 9.1 requires monitoring, measurement, analysis, and evaluation of both the AI systems and the AIMS processes around them; Clause 9.2 requires a structured internal audit programme, conducted at planned intervals, to verify the AIMS conforms to the standard and to the organization's own requirements; and Clause 9.3 requires top management to formally review the AIMS, considering performance data, risk status, and audit results, at planned intervals.
It's worth being precise about what Clause 9 does and doesn't specify. It governs the audit and review process itself, how often audits happen, who plans them, what management review must consider, rather than enumerating the substantive checklist (data lineage, bias testing, security controls, and so on). That substantive scope is spread across Clause 6 (risk assessment and treatment), Clause 8 (operational planning and control), and Annex A's controls (for example, A.7 on data resources and A.5 on AI system impact assessment). Organizations seeking or maintaining ISO/IEC 42001 certification must run internal audits against all of these, then feed the results into Clause 9.3 management review and Clause 10 continual improvement.
Local Law 144 of 2021 was enacted on December 11, 2021, with final implementing rules adopted by the NYC Department of Consumer and Worker Protection (DCWP) on April 6, 2023, and enforcement effective July 5, 2023. It requires any employer or employment agency using an 'automated employment decision tool' (AEDT) for a role connected to New York City to commission an annual, independent bias audit; that audit must calculate selection rates and impact ratios (using the four-fifths rule as the disparate-impact benchmark) broken out by race/ethnicity, sex, and intersectional categories. Employers must publish a summary of the results on their careers page and give candidates at least ten business days' notice before an AEDT is used. It is widely described as the first law in the United States to mandate independent bias audits of AI-driven hiring tools.
A December 2, 2025 audit by the New York State Comptroller (Thomas DiNapoli) examined how well DCWP has actually enforced the law between July 2023 and June 2025, and found significant gaps. Auditors made 12 test calls to NYC's 311 system to file an AEDT complaint; only 3 were correctly routed to DCWP, 8 were misdirected to the state Department of Labor, and one was sent back to the employer. Separately, DCWP's own compliance review of 32 employers' and vendors' websites identified just one likely instance of non-compliance, while the Comptroller's auditors, reviewing the same 32 websites, found 17 instances of potential non-compliance. The audit also found no ongoing public education effort after the law's initial rollout. It issued 13 recommendations, including fixing complaint routing, using existing city technology resources for compliance reviews, and proactively identifying non-compliance rather than relying only on complaints. The practical lesson for governance teams: an audit mandate with strong disclosure requirements can still produce compliance on paper without compliance in practice if the enforcement arm behind it is under-resourced.
What is an AI audit?
An AI audit is an independent examination of an AI system and its governance to check whether it performs as intended, is fair, is safe, and complies with relevant laws and internal policy. It is typically performed by a party independent of the team that built or operates the system.
How is an AI audit different from AI testing or AI monitoring?
Testing is done by the builders before and during deployment; monitoring is continuous oversight usually run by a second-line risk or compliance function; audit is periodic and independent of both, reporting findings to a board or audit committee rather than to the team being reviewed. This maps to the Institute of Internal Auditors' Three Lines Model.
What does NYC Local Law 144 require for AI bias audits?
Employers using an automated employment decision tool for a NYC-connected role must commission an annual independent bias audit calculating selection rates and impact ratios by race/ethnicity and sex (including intersectional categories), publish a summary of results on their website, and give candidates at least 10 business days' notice before the tool is used. It took effect for enforcement on July 5, 2023.
Has NYC Local Law 144 actually been enforced?
Enforcement has been weak so far. A December 2, 2025 audit by the New York State Comptroller found DCWP's complaint-routing process largely broken (only 3 of 12 test calls were correctly routed) and that DCWP's own compliance sweep missed most of the violations the Comptroller's auditors found in the same sample of employer and vendor websites. The Comptroller issued 13 recommendations to fix this.
What does ISO/IEC 42001 require for internal audits of AI systems?
Clause 9 of ISO/IEC 42001:2023 requires organizations to monitor and measure AIMS performance (9.1), run a structured internal audit programme at planned intervals to verify conformance (9.2), and have top management formally review the AIMS, including audit results, at planned intervals (9.3). The substantive scope being audited, data quality, bias, security, and so on, is defined elsewhere in the standard, in Clause 6, Clause 8, and Annex A.
Can an AI audit be conducted internally, or must it be independent?
It depends on the framework. NYC Local Law 144 defines an 'independent auditor' as someone not involved in using or developing the tool, which can include internal staff if a different team handles the audit. ISO/IEC 42001's internal audit (Clause 9.2) can likewise be performed by the organization itself or by an external party. In both cases, the assurance value of the audit depends on genuine separation from the people who built or operate the system, not just a different job title.
Last reviewed July 2026