The vendor due diligence gap
Organisations procuring AI systems from vendors frequently underinvest in pre-procurement due diligence and overestimate the protection provided by vendor contracts. The result is organisations that have deployed AI systems they do not fully understand, cannot adequately govern, and cannot demonstrate are compliant with their regulatory obligations, because they relied on vendor assurances rather than verifiable evidence.
The EU AI Act has changed this calculus significantly. Under the Act, deployers of AI systems, the organisations that put AI into use, regardless of whether they built it, bear independent legal obligations. These obligations include implementing human oversight mechanisms, conducting fundamental rights impact assessments in some contexts, monitoring AI system performance, and suspending use where serious risks are detected. These obligations cannot be contracted away to a vendor.
Effective AI vendor due diligence is the process by which organisations gather sufficient verified information about an AI system to satisfy their own governance and compliance obligations, and to make an informed procurement decision. It is not a box-ticking exercise, it is the foundation for defensible AI governance.
Pre-procurement: the questions that must be asked
System classification and regulatory status
Is this AI system classified as high-risk under the EU AI Act or any other applicable regulation? If so, has the vendor conducted the required conformity assessment, and can they provide the conformity assessment documentation? Is the system registered in the EU AI database if required? What regulatory approvals or certifications does the system hold, and in which jurisdictions?
Many vendors have not conducted this analysis and will provide vague assurances rather than documented answers. The inability or unwillingness to provide clear answers to these questions is itself a material finding.
Technical documentation
Under the EU AI Act, providers of high-risk AI must supply deployers with the technical documentation necessary to assess compliance. This documentation must include: the intended purpose and capabilities of the system; the training, validation, and testing datasets used; performance metrics in the intended deployment environment; known risks and limitations; and the human oversight measures built into the system.
Require this documentation before signing. Review it with appropriate technical and legal expertise. Vendors who describe their system's methodology as proprietary and refuse to provide technical documentation for commercially sensitive systems are not in compliance with their EU AI Act obligations as providers, and procuring from them creates compliance risk for your organisation as a deployer.
Training data
Where did the training data come from? Does the vendor have clear rights to use this data for this purpose? Is the training data representative of the population your organisation will apply the system to? What data quality controls were applied? Has the training data been tested for bias, and what were the results?
These questions matter for governance, and they matter for performance. An AI system trained on data that does not represent your deployment context will perform worse than advertised, in ways that may not be immediately apparent.
Bias testing and fairness evidence
Has the system been tested for bias and disparate impact? If so: what testing methodology was used, against which protected characteristics, using which fairness metrics, and what were the results? How does the vendor define "fair" for this system, and is that definition appropriate for your use case?
Vendor claims of "unbiased AI" without supporting methodology and results are marketing, not governance evidence. Require the underlying testing documentation, not the summary assertion.
Data processing and model training rights
The standard terms of many AI vendor agreements include broad rights to use customer data to train, improve, or validate the vendor's models. This means your organisation's operational data, potentially including personal data, confidential business information, or regulated data, could be used by the vendor to improve AI systems sold to competitors.
Review data processing agreements specifically for: rights granted to the vendor to use your data for model training or improvement; whether your data is used in aggregated or anonymised form or in identifiable form; data residency requirements and whether data leaves your jurisdiction; and your rights to audit vendor data use.
Ongoing obligations and change management
AI systems change over time, model updates, data updates, capability changes. What is the vendor's process for notifying customers of material changes to the AI system? What testing is conducted before deploying updates? What is your right to opt out of updates that materially change system behaviour? What happens to your governance documentation when the system changes?
Contract requirements for AI procurement
Standard technology vendor contracts are typically inadequate for AI procurement. At minimum, AI procurement contracts should address: technical documentation access and update obligations; data processing restrictions including explicit prohibition on using your data to train third-party models; notification requirements for material system changes; audit rights enabling your organisation to independently verify compliance; liability allocation for AI failures that cause harm; and termination rights if the vendor cannot demonstrate compliance with applicable AI regulations.
Legal review of AI procurement contracts should be approached with the same seriousness applied to technology contracts in regulated contexts. The regulatory exposure created by an inadequate AI vendor contract is material.
Ongoing vendor governance
AI vendor due diligence is not a one-time pre-contract exercise. Vendor practices change, models are updated, regulatory requirements evolve, and performance in deployment may diverge from pre-contract representations.
Effective ongoing vendor governance requires: annual review of vendor AI governance documentation; monitoring of vendor regulatory compliance status, particularly for vendors in regulated deployment contexts; performance monitoring in your deployment environment, not just reliance on vendor-reported metrics; a process for reviewing vendor changes before deployment; and a relationship management approach that maintains the vendor relationship necessary to raise governance concerns constructively.
The organisations that will face the most significant AI governance problems in the next five years are those that treated AI procurement as a standard technology purchase and vendor management as a commercial function rather than a governance function. The regulatory environment has changed what is required.
