Someone Will Have to Explain Your Model Under Oath. Decide Now Who.
Every AI governance programme is ultimately a bet about a room it never mentions: a deposition, three years after the decision, where a person has to say what the system did and why. The vendor will not be there. The model has been retrained twice. What you can prove is whatever you happened to write down at the time.
The Compliance Artifact Nobody Designs For
Companies build AI documentation for regulators: impact assessments, model inventories, bias audits, incident logs. That work is real and it is not the same work as being able to defend a single decision on a single day to a single person. Regulatory documentation describes a system in general. Litigation asks what happened to this claimant, on this date, with these inputs.
The gap shows up in a predictable place. An employer sued over an AI-screened rejection can produce a beautiful bias audit covering the model as a whole. Asked what score this candidate received, which model version produced it, and what the system was told about them, the answer is often that the scores are not retained, the version is not recorded, and the input snapshot was assembled at request time from systems that have since changed. The audit proves the tool was studied. It does not prove what the tool did.
Our guide to records retention for algorithmic hiring covers what to keep in that setting. This piece is about who then stands up and explains it, and what makes that testimony survive a challenge.
What Rule 702 Demands After 2023
The amended rule makes explicit that the proponent must establish admissibility by a preponderance, and that the opinion must reflect a reliable application of reliable methods to the facts of the case. The reliability inquiry is not satisfied by an expert's credentials, and the classic Daubert factors — testability, error rate, peer review, general acceptance, standards controlling operation — translate awkwardly to a proprietary model.
- Testability. Can the opinion be checked? An expert who cannot run the system, or run the relevant inputs against a preserved version of it, is offering an inference about a black box rather than an examination of it.
- Error rate. Cuts both ways and is frequently the strongest ground available to you — validation results, holdout performance, and monitoring data are a genuine known error rate, which most challenged forensic methods have never had. The catch is that the rate must be for your deployment on your population, not the vendor's benchmark on theirs.
- Standards controlling operation. Where governance documentation pays off directly. A documented risk management process, change control, and monitoring regime is exactly what this factor asks for, and it is the reason a maintained model inventory has litigation value beyond its compliance value.
- General acceptance. Usually about the method rather than the product — gradient boosting on tabular features is uncontroversial; a bespoke composite score with hand-tuned weights and no validation study is a different conversation.
The Vendor Problem
The people who genuinely understand the model do not work for you, and they have strong reasons to stay out of it. Testimony about the product creates a transcript that plaintiffs in every other customer's case will read. It puts trade secrets in front of a protective order. It costs senior engineering time on a matter with no upside for the vendor. So the default answer is no.
That answer is negotiable only in advance. The clause is short and unglamorous: on reasonable notice and at defined rates, the vendor will provide a knowledgeable witness and the technical information reasonably required to respond to a claim arising from the system, subject to a protective order. Vendors resist it, and the resistance itself is informative — a supplier unwilling to stand behind the product in the one forum where the product's behaviour is examined is telling you how they view the risk. The wider clause set sits in our AI vendor contract guide.
Where you cannot get cooperation, plan around it. That means preserving more on your side — inputs, outputs, versions, configuration — precisely because you will have to explain the system from the outside, using its observable behaviour rather than its internals.
The Six Records That Make Testimony Possible
- 1. Version identity per decision. Which model, which weights, which configuration, which thresholds — recorded on the decision record, not inferable from a deployment timeline.
- 2. The input as the model saw it. Not a pointer to source systems that have since been updated. The actual feature payload, snapshotted.
- 3. The raw output. Score, ranking, probability, or reason codes — before the business rules that turned it into an action.
- 4. The human step, honestly recorded. Who reviewed, what they saw, how long they took, and whether they ever departed from the recommendation. A reviewer who has never once overridden the system is evidence against the oversight you claim to have.
- 5. Contemporaneous validation. Testing done before deployment and monitoring done during it, dated, with the results as found rather than as summarised.
- 6. The change log. Every retrain, threshold adjustment, and vendor update, with dates — so a decision can be tied to the system that actually made it.
None of these is exotic and all of them must exist before anyone sues. The asymmetry is brutal: creating them costs engineering hours during a build, and cannot be done at all afterwards.
The Preparation Failure That Recurs
The most common own goal is sending the wrong person. A data scientist designated as a corporate representative answers questions the way engineers talk — candidly, in probabilities, with enthusiastic acknowledgement of everything the model does poorly, and with speculation about components they do not own. Every one of those instincts is correct in engineering and damaging in a deposition. A representative testifies to the organisation's knowledge, on the noticed topics, and does not speculate. That is a skill, it is taught in preparation sessions, and companies that skip it hand the other side their best material.
Frequently Asked Questions
Does an explainability tool like SHAP or LIME satisfy the reliability requirement?
It helps and it is not sufficient on its own, and the distinction matters because these tools are frequently oversold internally. Post-hoc attribution methods are approximations with known instability — different methods, and sometimes the same method on different runs, can attribute a decision differently. An expert who presents an attribution plot as the reason a decision was made is vulnerable on cross. An expert who presents it as one line of evidence, alongside validation results, monitoring data, and the decision record, is on much firmer ground.
We use a general-purpose LLM rather than a trained classifier. Does that change the analysis?
It makes it harder, and companies deploying LLMs into consequential decisions rarely account for this. There is no error rate in the traditional sense, outputs are not deterministic unless you fix the sampling parameters, the model behind an API changes without your involvement, and the vendor will not give you the weights. What you can control becomes disproportionately important: pin the model version where the API allows it, log the full prompt and completion, fix and record generation parameters, and run your own periodic evaluation set so you have dated evidence of behaviour over time.
How long should we retain decision records?
Longer than the applicable limitation period, which is longer than most retention schedules assume. Employment discrimination, credit, and consumer claims each have their own clocks, some of which run from discovery rather than from the decision, and a class claim can reach back further than an individual one. The costly mistake is a schedule tuned to storage cost rather than to exposure — decision records are small, and the marginal cost of keeping them is trivial against the cost of a case you cannot document. Set the retention deliberately with counsel rather than inheriting a default.
Can the other side get our model source code?
Potentially, under a protective order, and the mechanics are well established from patent practice — attorneys'-eyes-only designation, a secure review room, no network access, printing limits, and a controlled expert list. Courts weigh the requesting party's need against the burden and will often order something narrower first, such as documentation, validation results, or the ability to run defined test inputs. Arguing for the narrower alternative is a much better posture than arguing for nothing.
Our system was bought as compliant and certified. Doesn't that resolve the reliability question?
It is evidence and it is not a defence. A certification tells a court that a process was followed against a standard, usually assessing the product in general rather than your deployment on your population. The questions that decide the case are downstream of that: whether the system was suitable for your use, configured correctly, monitored, and overseen by humans with real authority. Companies that treat a certificate as the answer tend to have stopped generating the evidence that would actually have answered it.
Run the Reconstruction Drill
Pick one adverse decision your systems made six months ago. Ask the team to produce the model version, the exact inputs, the raw output, the human review record, and the validation evidence current at that date. Give them a week.
Whatever they cannot produce is what your witness will not be able to say. That list is your actual litigation readiness gap, and it is far cheaper to close it now than to explain it later — the drill costs a week and tells you more than any tabletop exercise about a hypothetical claim ever will.