Test Data Is an Admission, Not a Shortcut
Local Law 144 lets an auditor substitute test data when historical data is insufficient. What it does not let you do is stay quiet about it — the public summary has to name the substitution and justify it, in a document your candidates and a plaintiff's lawyer can both read.
The default is the tool's own history
A Local Law 144 bias audit measures what an Automated Employment Decision Tool did to real people. The rules therefore start from historical data: the applicants or employees the tool actually scored, joined to their demographic categories and their outcomes, so the auditor can compute selection rates and impact ratios that describe your funnel rather than a laboratory.
This is where most audit projects stall, and the reason is mundane. Outcomes live in the ATS, scores live in the vendor system, and the EEO self-identification survey lives in a third place that was deliberately firewalled from recruiters. Nobody designed those systems to be joined. Budget the extraction, not the statistics — the statistics are a day of work once the table exists.
Three lawful data postures
Your own historical data
STRONGESTThe tool's prior use on your applicants, with your thresholds and your funnel. This is the audit that actually describes the tool as deployed, and the only one that cannot be attacked on scope grounds.
It is also the audit that finds real problems, which is a feature. A disparity you discover in an audit is cheaper than one a claimant discovers for you.
Pooled historical data from multiple employers
PERMITTEDAn audit may use historical data from more than one employer or employment agency that used the same AEDT. Vendor-level audits are built this way, and an employer may rely on one if its own data was in the pool or if it has never used the tool.
Ask two questions before relying on it: was our data included, and does the pool resemble our applicant population? A pool dominated by another industry produces ratios that say little about your funnel.
Test data
CONDITIONAL, DISCLOSEDPermitted only where there is insufficient historical data to produce a statistically significant audit. The published summary must state that test data was used and explain why historical data was insufficient.
That explanation is a public statement about the maturity of your compliance program. Reasonable for a new deployment; conspicuous in year three of using the same tool.
What the public summary has to carry
The summary is the artifact the world sees, and it is the artifact DCWP looks for first because it can be checked without opening an investigation. It has to include:
The date of the most recent bias audit
Which is what the one-year currency rule is measured against. An undated summary is functionally a missing summary.
The source and explanation of the data used
Historical or test, whose data, and — where test data was used — why historical data was insufficient.
The number of individuals assessed
Counts by category, so a reader can judge whether a ratio rests on nine people or nine thousand.
Selection or scoring rates by category
Sex, race/ethnicity in the EEOC categories, and the intersectional combinations of the two.
Impact ratios, including intersectional cells
Each category's rate divided by the rate of the most-selected category. Intersectional cells are where disparities that cancel out in the marginals become visible.
Excluded categories and their counts
Any category under 2% of the data may be left out of the ratio math, but the summary must name it and give its number of applicants and its rate.
The distribution date of the AEDT
The date the tool was put into use, which anchors the audit to a specific deployed version.
The summary must be posted on the employer's or agency's website in a clear and conspicuous place, or reachable by an active link labeled as the bias audit summary, and it stays up for at least six months after the tool was last used.
The Unknown category is the quiet failure mode
Demographic self-identification is voluntary, and in a lot of funnels a large share of candidates decline. Those candidates do not disappear from the audit — they land in an Unknown group, and if that group is large the ratios computed on the remainder describe a minority of your applicants.
An auditor who does the work properly will flag the limitation rather than quietly drop the rows. If your response rate is poor, the fix is upstream: ask for self-identification at a moment in the flow where candidates are willing to answer, and make clear it is separated from the screening decision. That is a funnel design change, not an audit finding, and it has to happen months before the audit to help.
Publishing a bad ratio is not the worst outcome
Nothing in Local Law 144 bans a tool that shows adverse impact. The law is a disclosure regime; it converts a private statistical fact into a public one. The exposure that follows comes from the older law — Title VII, the New York State and City Human Rights Laws — where a published impact ratio below 0.80 that you kept using without a job-relatedness justification is the plaintiff's opening exhibit. The defensible posture is not a flattering summary. It is an honest one, paired with a documented record of what you changed after reading it.
Is your audit summary actually reachable?
"Clear and conspicuous" fails when the summary is a scanned PDF with no text layer, a link only reachable by hover, or a page no crawler and no screen reader can parse. Scan the page free and see what a candidate or an investigator would find.
Scan Your Compliance Page for Free →Frequently Asked Questions
What is historical data under Local Law 144?
Data from the AEDT's actual prior use — the real applicants or employees it scored, with their outcomes and demographic categories. The default rule is that the bias audit runs on that data, because the point of the audit is to measure how the tool behaved on real people rather than how it behaves on a constructed sample.
When can an auditor use test data instead?
Only when there is insufficient historical data to conduct a statistically significant audit. That is the condition, and it is the auditor's judgment to make, not the employer's preference. When test data is used, the published summary must say so and explain why historical data was insufficient.
We just bought the tool and have never run it. What do we do?
This is the ordinary path into test data — a brand-new deployment has no history to audit. The other path is the vendor's pooled historical data across its customers, which is permitted for a new deployment and is generally a stronger basis than a synthetic sample. Ask the vendor for a pooled audit before you settle for test data.
Can several employers share one bias audit?
Yes. An audit may rely on historical data pooled from multiple employers or employment agencies that used the same AEDT, which is how vendor-level audits work. An employer may rely on such an audit if its own data was included in the pool, or if it has never used the tool before.
What is the 2% rule about small categories?
A category that represents less than 2% of the data being audited may be excluded from the impact-ratio calculations — but the summary must identify the excluded categories and give the number of applicants and the scoring rate or selection rate for them. Exclusion is a disclosure, not a deletion.
Does a bad impact ratio mean we have to stop using the tool?
Local Law 144 sets no threshold that bans a tool. It compels publication. The consequence of a poor ratio is that you have published a document quantifying adverse impact — which becomes evidence in a discrimination claim under federal, state, or city law if you keep using the tool without a defensible justification.