Your Training Set Needs a Paper Trail, Not a Bucket Name
Most high-risk AI documentation projects start with the model card and the risk register. The article that fails audits is the quiet one before them: the data and data-governance requirement, which asks where every training, validation and test set came from, what was done to it, and what you found when you examined it for bias.
The Article That Cannot Be Written Retroactively
Nearly every other obligation attached to a high-risk system can be produced after the fact by a competent team with a quarter to spend. A quality management system can be written down. A risk register can be reconstructed. Human-oversight measures can be designed and retrofitted into the interface. The data article is different in kind, because it asks questions about events that already happened — who collected this, under what instruction, from where, with what exclusions — and if nobody recorded them at the time, no amount of later effort recovers the answer.
That asymmetry is why the dataset section is the one that stalls conformity work. The honest version of most first drafts reads: an export from the production database, filtered to rows since some date, labelled by a team that has since been reorganised, plus a licensed corpus whose terms nobody has reread. Every word of that is normal engineering practice and none of it answers the statutory questions.
The Seven Questions Your Record Has to Answer
- Design choices. Why this data represents the problem, and what alternatives were rejected.
- Origin and collection. Where the data came from, by what process, and under what lawful basis or licence.
- Preparation. Labelling, cleaning, enrichment, aggregation — including labeller instructions and inter-annotator disagreement.
- Assumptions. What the data is presumed to measure, stated plainly enough that a reviewer can disagree with it.
- Adequacy. Whether volume and suitability match the intended purpose, with the reasoning shown.
- Bias examination. What you looked for, how you looked, and what you found — including nothing.
- Mitigation and gaps. Measures taken, what remains missing, and how the residual is handled in deployment.
Write them as one document per dataset version, not one per system. Systems get retrained; datasets get amended; an unversioned record describes a system that no longer exists, and the mismatch between the record and the deployed artefact is exactly what an assessment looks for first.
Representativeness Is Decided by Your Intended Purpose
The Act asks for datasets that are relevant, sufficiently representative and, to the best extent possible, free of errors and complete in view of the intended purpose, with attention to the geographical, contextual, behavioural and functional setting of use. The trap is treating representativeness as a universal demographic target. It is not: it is a comparison between the population you sampled and the population you told everyone the system is for.
That makes the intended-purpose statement a load-bearing document rather than marketing copy. A narrow purpose is far easier to evidence — one sector, one language, one regulatory context — and the commercial instinct to write the broadest possible scope silently imports a representativeness obligation across every market named. Teams that have been through an assessment tend to narrow the purpose and add markets deliberately, with the dataset work attached to each addition.
The Special-Category Deadlock, and How the Act Resolves It
Every bias examination runs into the same internal argument. Privacy counsel says the organisation does not collect ethnicity, religion, health status or sexual orientation. The data science team replies that you cannot measure disparate impact on an attribute you refuse to hold. Both are correct under GDPR alone, and the argument usually ends with a proxy analysis that satisfies nobody.
The Act contemplates a narrow exception: processing special categories of personal data where strictly necessary for bias detection and correction in high-risk systems, with safeguards attached — no alternative with synthetic or anonymised data, technical limits on reuse, pseudonymisation and state-of-the-art security, logged and restricted access, no onward transmission, and deletion when the bias has been corrected or the retention period lapses. Read those as build requirements. The compliant shape is a separate evaluation dataset with its own access control and an explicit deletion trigger, never joined into the training pipeline or the production feature store.
Where Deployers and Fine-Tuners Get Pulled In
The data article is written for providers, which leads deploying organisations to file it as somebody else's problem. Three routine acts move the role. Putting your own name or trademark on a high-risk system already on the market. Making a substantial modification to it. And changing its intended purpose so that a system becomes high-risk in your hands. Fine-tuning a general-purpose model on your own customer data for a high-risk use is the common path, and the tuning corpus is then a training dataset with the full record attached to it.
Deployers who never cross that line still inherit a smaller, real duty: where they control input data, they must ensure it is relevant and sufficiently representative in view of the system's intended purpose. In a hiring tool that means the candidate pool you feed it; in a credit tool, the bureau extract you select. It is a narrower obligation than the provider's, and it is the one most likely to be discovered missing because nobody thought the buyer had any data obligations at all.
Questions Compliance and ML Teams Ask
Our model was trained before the obligations applied. Do we have to reconstruct the record?
The transitional rules treat legacy high-risk systems differently from new ones, and the detail of your situation — when the system was placed on the market, whether it has been substantially modified since, and whether it is a public-sector deployment — decides which regime you sit in. What is common to every version of the answer: the moment you retrain, materially change the intended purpose, or ship a substantial modification, you are documenting a current dataset and the legacy argument stops helping. Most teams find it cheaper to write the record for the next training run properly than to litigate the status of the previous one, and doing so converts an open question into a dated artefact.
How detailed does the labelling description have to be?
Detailed enough that a reader can identify where systematic label error would enter. In practice that means the actual instructions given to labellers, the population the labellers were drawn from, the adjudication rule for disagreement, and the measured disagreement rate. The last one is the most diagnostic and the most commonly absent: a task with high inter-annotator disagreement has an unstable target, which is a data-quality finding in its own right and frequently explains a bias result that everyone else attributes to the model. If labelling was outsourced, the instruction document is the artefact to secure in the contract, because the vendor will otherwise treat it as proprietary methodology.
Does synthetic data reduce the burden?
It changes the questions rather than removing them. A synthetic set still has design choices, a generation process, assumptions and limitations, and the generator itself was trained on something whose provenance travels forward. Synthetic data is genuinely useful for the special-category problem, because the exception for processing real special-category data is conditioned on the task not being achievable by other means — so a credible synthetic or anonymised route can keep you out of that regime entirely. It is much weaker as a representativeness argument: a generator trained on an unrepresentative corpus produces unrepresentative data at greater volume, which reads as a larger dataset with the same gap.
What does an assessor actually ask to see first?
The mapping between the dataset versions and the deployed model version, because it is the fastest way to test whether the documentation describes reality. If the record names a dataset snapshot and the serving artefact was built from a different one, everything downstream is in question. After that: the bias examination and what was done with its findings, the intended-purpose statement read against the markets you sell in, and the labelling instructions. A tidy narrative with no artefacts behind it is a poorer position than a messy record with real evidence attached, and the second is what a version-controlled pipeline produces for free.
Where should the record physically live?
In the repository that builds the model, versioned alongside the pipeline, and rendered into the technical documentation rather than maintained separately in a document store. The reason is drift: a dataset description kept in a compliance folder is updated when somebody remembers, while one that sits next to the training code is updated by the person changing the data, at the time of the change. That is also what makes the record cheap to produce at the moment an authority or an enterprise customer asks — and enterprise procurement is asking for it well ahead of any regulator, which is the commercial reason to do this work before the compliance reason bites.
Start With the Dataset You Are About to Build
Do not open the archaeology project first. Take the next training run, write the seven answers before the data is assembled rather than after, and version the document with the pipeline. One well-evidenced current dataset is worth more in an assessment than a reconstructed history of four old ones.
Then work backwards only as far as a system you still sell requires — and let the ones you are sunsetting be sunset.