AI data governance: what the law requires of the data used to train, test, and run AI systems

What does the law require of AI data governance?

No single US or EU law regulates all data used in AI. The duties come from five sources: the EU AI Act’s data rules for high-risk systems and its training-content summary for general-purpose models, the GDPR and US privacy statutes for personal data, state transparency laws such as California AB 2013, sector rules such as HIPAA and the FCRA, and FTC orders that require deletion of models built on unlawfully used data.

Read together, they ask a company to know where each dataset came from, to have a legal right to use it for training, to test it for bias and fitness for the intended use, to document all of that before release, and to be able to delete the data and what was built from it.

AI data governance requirements by law, as of October 26, 2026

Requirement	Law	Who it binds	What to keep on file
Data governance practices for training, validation, and testing sets	EU AI Act Article 10(2)	Providers of high-risk AI systems, from December 2, 2027 (Annex III) or August 2, 2028 (Annex I Section A)	Datasheets on provenance, scope, main characteristics, how data was obtained and selected, labelling, and cleaning (Annex IV, point 2(d)); kept for 10 years after the system is placed on the market or put into service (Article 18)
Relevant, sufficiently representative, and, to the best extent possible, free of errors and complete data	EU AI Act Article 10(3)	Same providers	Statistical properties for the persons or groups the system is intended to be used on
Special-category data processed only where strictly necessary for bias detection and correction	EU AI Act Article 4a (formerly Article 10(5))	High-risk providers (4a(1)); other providers, deployers of high-risk systems, and deployers of other systems (4a(2))	GDPR records of processing stating why the processing was strictly necessary and why other data would not do; deletion once the bias is corrected or the retention period ends, whichever comes first
Relevant and sufficiently representative input data	EU AI Act Article 26(4)	Deployers of high-risk systems, to the extent they control the input data	Input data specifications and checks
Public summary of training content	EU AI Act Article 53(1)(d)	General-purpose AI model providers, including open-source providers	Published summary on the AI Office template; training, testing, and validation data description for the AI Office (Annex XI, point 2(c))
Lawful basis, compatible purpose, special-category limits, automated decisions	GDPR Articles 5(1)(b), 6, 9, 22, 35	Controllers subject to the GDPR	Legitimate interest assessment, compatibility assessment, DPIA, documentation supporting any claim that a model is anonymous
Training data documentation on the developer’s website	California AB 2013 (Civil Code 3110 to 3111)	Developers of generative AI systems or services released on or after January 1, 2022 and made publicly available to Californians	The posted 12-item summary, updated before each new release or substantial modification
Risk assessment before processing personal information to train certain systems	CCPA regulations, 11 CCR 7150(b)(6)	Businesses subject to the CCPA	The risk assessment; attestations and summaries for 2026 and 2027 assessments due April 1, 2028
Categories of training data disclosed to deployers	Colorado SB 26-189, C.R.S. 6-1-1702(1)	Developers of covered ADMT, from January 1, 2027	Deployer documentation; compliance records for at least 3 years
De-identification before health data is used outside the Privacy Rule	HIPAA, 45 CFR 164.514(b)	Covered entities and business associates	Expert determination methods and results, or the safe harbor identifier removal
Permissible purpose for consumer report data; adverse action notice	FCRA, 15 U.S.C. 1681b(f) and 1681m(a)	Any person that uses or obtains a consumer report	Permissible purpose certification; adverse action notices and credit score disclosures

What does EU AI Act Article 10 require?

Article 10 requires that a high-risk AI system that trains models with data be developed on training, validation, and testing data sets that meet the quality criteria in Article 10(2) to (4) and in Article 4a(1) whenever such data sets are used (Article 10(1), as amended). Under Article 10(2), those data sets must be subject to data governance and management practices appropriate for the system’s intended purpose, concerning in particular:

• the relevant design choices
• data collection processes and the origin of data, and, for personal data, the original purpose of the data collection
• data preparation, such as annotation, labelling, cleaning, updating, enrichment, and aggregation
• the assumptions about what the data are supposed to measure and represent
• an assessment of the availability, quantity, and suitability of the data sets needed
• examination for possible biases likely to affect health and safety, have a negative impact on fundamental rights, or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations
• appropriate measures to detect, prevent, and mitigate the biases identified
• the identification of data gaps or shortcomings that prevent compliance, and how they can be addressed

Article 10(3) sets the quality standard: the data sets must be relevant, sufficiently representative, and, to the best extent possible, free of errors and complete in view of the intended purpose, with appropriate statistical properties, including, where applicable, for the persons or groups on whom the system is intended to be used. Under Article 10(4), data sets must also take into account, to the extent the intended purpose requires, the characteristics or elements particular to the specific geographical, contextual, behavioural, or functional setting in which the system is intended to be used. For a system that does not train models, Article 10(2) to (4) and Article 4a(1) apply only to its testing data (Article 10(6)).

To the extent a deployer of a high-risk system controls the input data, it must ensure that input data is relevant and sufficiently representative in view of the system’s intended purpose (Article 26(4)).

Article 10 sits in Chapter III, Section 2, which applies from December 2, 2027 for stand-alone Annex III systems and from August 2, 2028 for AI in Annex I Section A regulated products (Article 113(c), as amended). The high-risk classification guide explains which systems are covered.

How did the Digital Omnibus change Article 10?

The Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since July 27, 2026, deleted Article 10(5) and re-enacted its exception as a new Article 4a. Article 10(2) to (4) are unchanged in the consolidated text dated July 27, 2026.

Under Article 4a(1), to the extent strictly necessary to ensure bias detection and correction under Article 10(2)(f) and (g), providers of high-risk systems may exceptionally process special categories of personal data, subject to appropriate safeguards for fundamental rights, and in addition to the GDPR, only if all of the following conditions are met:

• the bias detection and correction cannot be effectively fulfilled by processing other data, including synthetic or anonymised data
• the data are subject to technical limitations on re-use and state-of-the-art security and privacy-preserving measures, including pseudonymisation
• the data are secured and protected, with strict controls and documentation of access, so that only authorised persons with appropriate confidentiality obligations have access
• the data are not transmitted, transferred, or otherwise accessed by other parties
• the data are deleted once the bias has been corrected or the data has reached the end of its retention period, whichever comes first
• the GDPR records of processing activities include the reasons why the processing was strictly necessary to detect and correct biases, and why that objective could not be achieved by processing other data

Under Article 4a(2), providers and deployers of other AI systems and models, and deployers of high-risk systems, may also exceptionally process special-category data, but only where strictly necessary for bias detection and correction in view of possible biases likely to affect health and safety, have a negative impact on fundamental rights, or lead to discrimination prohibited under Union law, and only if all of the Article 4a(1) conditions and safeguards are applied. The paragraph states that it does not create any obligation to conduct bias detection and correction.

Recital 9 of the Omnibus ties the legal basis to Article 9(2)(g) of the GDPR, substantial public interest, and says that, so providers of high-risk systems can lawfully undertake bias detection and correction in preparation for compliance with Article 10(2)(f) and (g), the Article 4a legal basis “should apply from the date of entry into application” of the AI Act. Article 4a sits in Chapter I, which has applied since February 2, 2025 (Article 113(a)), and the Omnibus sets no separate date for it, so Article 4a has applied since the Omnibus entered into force on July 27, 2026.

What must general-purpose AI model providers disclose about training data?

Since August 2, 2025, a provider of a general-purpose AI model must draw up and make publicly available a sufficiently detailed summary about the content used to train the model, according to a template provided by the AI Office, which the Commission published with an explanatory notice (Article 53(1)(d)). The open-source exemption in Article 53(2) covers only the documentation duties in Article 53(1)(a) and (b), so a provider that releases a model under a free and open-source license must still publish the summary. Providers of models placed on the market before August 2, 2025 have until August 2, 2027 to comply (Article 111(3)). The EU AI Act guide covers the rest of the general-purpose model regime.

How does the GDPR apply to AI training data?

Every training, fine-tuning, or evaluation set that contains personal data needs a lawful basis under Article 6, must respect purpose limitation under Article 5(1)(b), and, if it holds special categories of data, needs an Article 9 exception.

The EDPB set out how supervisory authorities should approach these questions in Opinion 28/2024, adopted December 17, 2024. Three of its points matter here:

• Anonymity is case by case. A model trained on personal data cannot, in all cases, be considered anonymous. For a model to be anonymous, both the likelihood of extracting personal data about people whose data was used in training and the likelihood of obtaining such data from queries should be insignificant.
• Legitimate interest needs the three-step test. A controller relying on legitimate interest must identify a lawful, clearly articulated, real and present interest, show the processing is necessary with no less intrusive way to pursue it, and show the interest is not overridden by data subjects’ rights, taking account of their reasonable expectations and of whether the data was publicly available.
• Unlawful training can follow the model. A controller deploying another company’s model should assess whether the model was developed by unlawfully processing personal data.

Purpose limitation decides whether customer data collected for one service can train a model. Unless the new use rests on consent or a qualifying law, the controller must weigh, among other factors, the link between the purposes, the context of collection, the nature of the data, the consequences for data subjects, and safeguards such as encryption or pseudonymisation (Article 6(4)).

Article 22 reaches the output side. It gives a data subject the right not to be subject to a decision based solely on automated processing, including profiling, that produces legal effects or similarly significantly affects them, with exceptions where the decision is necessary for entering into or performing a contract with the data subject, is authorised by Union or Member State law that lays down suitable safeguards, or is based on the data subject’s explicit consent. A controller relying on the contract or consent exception must implement suitable measures to safeguard the data subject’s rights, at least the right to obtain human intervention, to express their point of view, and to contest the decision (Article 22(3)). None of these decisions may be based on special categories of data unless explicit consent or substantial public interest under Article 9(2)(a) or (g) applies and suitable safeguards are in place (Article 22(4)).

The GDPR text has not changed for AI. The Commission’s November 19, 2025 proposal, COM(2025) 837, would add an Article 88c on processing personal data for the development and operation of AI systems and models; as of October 26, 2026, the Publications Office listed no adopted act amending the GDPR on this point.

What do California and Colorado require?

California AB 2013 requires the developer of a generative AI system or service released on or after January 1, 2022 and made publicly available to Californians to post, on or before January 1, 2026 and before each later release or substantial modification, documentation on its website about the data used to train the system, including a high-level summary of the datasets (Civil Code 3111). The statute lists 12 items the summary must include, among them the sources or owners of the datasets, the number and types of data points, whether the datasets include copyrighted, purchased or licensed, or personal information, any cleaning or other modification, the collection period, and whether synthetic data generation was used. xAI sued; the Central District of California denied a preliminary injunction on March 4, 2026, and the appeal (Ninth Circuit No. 26-1591) was pending as of October 2026, with oral argument set for November 18, 2026 on the Ninth Circuit’s calendar. The law is enforceable while the appeal proceeds.

The CPPA’s regulations add a training trigger. A business subject to the CCPA must conduct a risk assessment before processing consumers’ personal information that it intends to use to train ADMT for a significant decision concerning a consumer, or to train a facial-recognition, emotion-recognition, or other technology that verifies a consumer’s identity or conducts physical or biological identification or profiling (11 CCR 7150(b)(6)). The California AI laws guide covers the ADMT duties that apply from January 1, 2027.

Colorado SB 26-189 applies to consequential decisions made on or after January 1, 2027. A developer of covered ADMT must give each deployer, among other items, a description of the categories of data, including personal data, used to train the system, to the extent known (C.R.S. 6-1-1702(1)). A federal court order in X.AI LLC v. Weiser bars the Attorney General from initiating enforcement, including an investigation, for violations occurring on or before 14 days after the court rules on xAI’s preliminary injunction motion; as of October 26, 2026, that motion had not been filed. See the Colorado guide.

What happens when a model is trained on data the company should not have used?

In the United States, the FTC has ordered companies to delete the models and algorithms, not only the data. Each order below settled allegations; none was a court finding of liability.

• Everalbum (proposed January 11, 2021; finalized May 7, 2021). The order requires deletion of face embeddings derived from photos of users who did not give express consent and of any facial recognition models or algorithms developed with users’ photos or videos.
• WW International and Kurbo (court order entered March 3, 2022; announced March 4, 2022). In a COPPA case the Department of Justice filed for the FTC, the court-approved order requires deletion of personal information collected from children under 13 through the Kurbo program unless a parent gave verifiable consent, destruction of any models or algorithms developed in whole or in part with children’s personal information collected through the program, and a $1.5 million penalty, and requires a retention schedule under which children’s personal information is deleted no more than one year after the user last tracked food, weight, or activity.
• Rite Aid (complaint December 19, 2023; stipulated court order entered February 26, 2024). The FTC alleged the retailer deployed facial recognition without reasonable safeguards. The order bars Rite Aid from using facial recognition for security or surveillance purposes for five years and requires deleting the photos and videos of consumers collected for its facial recognition system and any data, models, or algorithms derived in whole or in part from them, and instructing third parties that received them to delete them too.

A company that records which datasets trained which models knows what a deletion order would reach.

Where do sector rules add data duties?

• HIPAA. Health information is de-identified only by expert determination, in which a qualified person finds the risk of re-identification very small and documents the methods and results, or by removing 18 listed identifiers with no actual knowledge that the remainder could identify someone (45 CFR 164.514(b)). Information de-identified that way falls outside the Privacy Rule (164.502(d)(2)); see the healthcare guide and the hospitals page.
• GLBA. For banks, the interagency information security standards, and for credit unions, NCUA’s guidelines for safeguarding member information, apply when an AI tool or vendor handles customer or member information. Financial institutions under FTC jurisdiction follow the FTC Safeguards Rule, which requires them to take reasonable steps to select service providers capable of maintaining appropriate safeguards, require those safeguards by contract, and periodically assess the providers (16 CFR 314.4(f)). The banks and credit unions page covers the service provider duties.
• FCRA. A person may not use or obtain a consumer report for any purpose unless it is a purpose the statute authorizes and the user has certified it (15 U.S.C. 1681b(f)). Any person that takes adverse action based in whole or in part on information in a consumer report must give the consumer notice of the action; any numerical credit score it used; the consumer reporting agency’s contact details and a statement that the agency did not make the decision; and notice of the consumer’s rights to a free copy of the report and to dispute its accuracy or completeness (1681m(a)). For hiring tools, see the employment guide.

How do NIST AI RMF and ISO/IEC 42001 treat data?

In NIST AI RMF 1.0, MAP 2.3 calls for documenting data collection and selection considerations, including availability, representativeness, and suitability. MEASURE 2.1 calls for documenting test sets and metrics, MEASURE 2.10 for examining privacy risk, and MEASURE 2.11 for evaluating fairness and bias.

ISO describes ISO/IEC 42001:2023 as setting requirements for an AI management system that include data governance and system lifecycle controls and that help an organization manage data quality. The control text is in the purchased standard. The governance frameworks guide compares the two.

What should an AI data governance checklist include?

• Inventory every dataset used to train, fine-tune, validate, test, or run each AI system, linked to the system inventory
• Record provenance for each dataset: source or owner, license or purchase terms, collection period, and the original purpose of any personal data
• Document the lawful basis for personal data in each use, with a legitimate interest assessment or Article 6(4) compatibility assessment where needed
• Examine data for bias and representativeness against the intended population, and record the measures taken and the gaps found
• Where special-category data is processed for bias detection, apply each Article 4a condition and record why other data would not do
• Set retention and deletion rules for training data, and keep a map of which models each dataset trained so deletion can reach them
• Require vendors to state the categories and sources of training data and whether they may train on your data

How this fits the other guides: the EU AI Act compliance checklist lists every provider and deployer duty by article; the agentic AI governance guide covers vendor contract terms on training rights.

This guide is general information as of October 26, 2026, not legal advice. Confirm obligations against the current text of each law and with counsel before acting on it.