What is the difference between a model card and a system card?

A model card documents a model in isolation: intended and out-of-scope uses, training-data characterization at whatever level the vendor is willing to give, evaluation results on public or internal benchmarks, and known limitations. A system card documents the deployed system around the model — the safety filters, retrieval layer, tool access, refusal behavior, human oversight points and mitigations that ship with the product. The distinction matters commercially because you almost never buy a bare model. You buy an application, and the model card describing its underlying base model can be entirely accurate while telling you nothing about the thing you deployed. Where a vendor offers only a model card for an application product, the correct read is that the layer you actually rely on is undocumented.

Is a model card audited or verified by anyone?

As a rule, no. Model cards are self-published by the developer, with no assurance standard, no independent examination and no defined scope of what must be included. That is not a criticism of the format — it was proposed as a transparency practice, not as an assurance artifact — but it does determine how much weight the document can carry. Compare it to a SOC 2 report, where an independent examiner tests stated controls against defined criteria, or to a conformity assessment, where the scope is set by regulation. A model card sits closer to a datasheet than to either. Use it to understand the product and to generate questions; do not file it as the evidence that a control exists.

Do benchmark results in a model card tell us anything about our use case?

Usually very little, for three reasons. Benchmarks measure aggregate performance on curated public tasks, and your use case is a narrow slice of a domain those tasks were not built to represent. Benchmark contamination is a persistent problem, since public test sets circulate on the open web and can appear in training corpora, which inflates scores without improving capability. And the metrics reported are typically accuracy-style aggregates that hide the distribution of errors, when what governs your risk is the tail — the rare confident wrong answer in a regulated decision. Treat published benchmarks as a coarse capability filter for shortlisting, then evaluate on your own held-out data with your own error costs before deployment.

Can we rely on a vendor's documentation for our own regulatory obligations?

You can incorporate it, and you cannot delegate to it. Where an obligation attaches to the party deploying a system and acting on its output, the deploying organization owes the assessment even when it did not build the model. Vendor documentation is an input to that assessment and often a necessary one, which is why the emerging pattern in AI regulation is to require upstream developers to supply downstream deployers with the information needed to comply. Two practical consequences: put the information-provision duty in the contract rather than relying on published pages that can change or disappear, and record in your own file how each piece of vendor documentation supports a specific element of your assessment. A folder of vendor PDFs is not an assessment.

The vendor updates the model but not the documentation. What should the contract say?

It should tie documentation to versions and give you notice on both. Three clauses do most of the work: current documentation must be provided for each model version you are entitled to use, with a stated update obligation on material change; you receive advance notice of a version deprecation or a material behavioral change to a version you are pinned to, long enough to revalidate; and prior versions remain available during a transition window. Without these, the ordinary sequence is that the vendor ships an improved model, your evaluation evidence silently becomes evidence about a model no longer in production, and you discover the gap when someone asks you to produce the current assessment. Documentation drift is not a rare failure — it is the default behavior of a fast-moving product.

Model Cards and System Cards 2026: What Buyers Should Demand and What They Actually Prove

What the Artifact Was Built For

Model cards were proposed as a transparency practice: a short structured document telling anyone considering a model what it was intended for, what it was evaluated on, and where it is known to fail. As a practice it was a genuine improvement over shipping a model with a README and a download link.

What has happened since is a category error in procurement. The card arrives in a diligence packet, gets stored in the vendor file, and closes the transparency line item in a governance checklist. The document has not changed; the weight placed on it has. A page written to help engineers choose a model is now being asked to serve as the organization's evidence that a system was assessed.

The gap is easiest to see by asking what a card would have to contain to answer a real compliance question — whether this system produces disparate outcomes for your population, in your configuration, at the version currently serving production — and then checking whether any card you have ever read contained it.

Four Questions the Card Structurally Cannot Answer

How does it perform on your population?

Cards report aggregate results on public or internal benchmarks constructed to be general.

Why the gap matters: Your exposure lives in a specific distribution — your customers, your document types, your dialects, your edge cases. Aggregate accuracy on a general benchmark is a capability signal and not a statement about your distribution, and the two routinely diverge in the direction that costs money.

What does the deployed system do, as opposed to the model?

The system prompt, retrieval layer, safety filters, tool permissions and fallback behavior are the vendor's product, and they are rarely documented in a card.

Why the gap matters: Almost every failure that reaches a customer is a system-layer failure: a filter that did not fire, a retrieval step that pulled the wrong record, an agent that took an action outside its intended scope. A card describing the underlying model is silent on all of it.

Which version is serving your traffic right now?

Cards are published per model version; production endpoints frequently route to a moving alias.

Why the gap matters: If your endpoint resolves to whatever is current, the documentation in your file may describe a model you stopped using months ago. Every downstream artifact built on it — your evaluation, your bias testing, your risk assessment — inherits the same staleness silently.

What is in the training data?

Training-data sections are typically characterized at a high level, or omitted, for competitive and litigation reasons.

Why the gap matters: You will not get a manifest and asking for one wastes leverage. What you can get is a characterization plus a warranty — source categories, treatment of opt-out signals, known claims — and the difference between a vendor who will stand behind a characterization and one who will not is the actual diligence signal.

Reading a Card Critically

The document is still worth reading closely. Four things to look for, all of which are more informative than the benchmark table:

The out-of-scope list. Where a vendor names uses the model is not suitable for, and your deployment appears on that list, you have a documented deviation from the vendor's stated intended use. That is a bad fact in a file and a straightforward one to catch at procurement.
The evaluation population. Look for what the evaluations were run on and in what languages. A model evaluated in one language and deployed against a multilingual customer base has an untested surface, and that is usually where the accuracy complaints originate.
The date and version. An undated card, or one that does not name a specific version, cannot support any claim about what is running now. Ask which endpoint the card corresponds to.
The limitations section's specificity. Generic cautions about hallucination are boilerplate. Specific, quantified failure modes indicate the vendor tested seriously and is willing to publish results that are not flattering — which is the strongest positive signal the format can carry.

The Disclosure Set to Require Instead

Ask for these at procurement, in the contract, with a version reference attached to each. The card is one line item in the set, not the set.

1. Scope and Identity

☐The exact model version or versions serving your account, and whether your endpoint is pinned
☐System-level documentation: prompts, filters, retrieval, tool access, fallback behavior
☐Intended-use and out-of-scope statements, checked against your actual deployment
☐The date each document was produced and the update obligation on material change
☐Whether any component is subcontracted, and to whom

2. Evidence of Behavior

☐Evaluation results for a use case resembling yours, not a general benchmark suite
☐The evaluation methodology, dataset construction and any contamination controls
☐Fairness or disparate-impact testing where the system touches a regulated decision
☐Known failure modes with frequency estimates, not adjectives
☐Red-teaming or adversarial testing summary and the date it was performed

3. Data and Rights

☐Training-data characterization by source category, backed by a warranty
☐Treatment of scraping opt-out signals and any known-infringing sources
☐Whether your inputs and outputs are used for training, fine-tuning or evaluation
☐Retention for abuse monitoring, including whether humans review retained content
☐Current subprocessor list with a notice obligation on change

4. Keeping It Current

☐Advance notice of deprecation for any version you validated against
☐Notice of material behavioral change to a pinned version
☐Prior-version availability during a defined revalidation window
☐An information-provision clause supporting your own regulatory assessments
☐A scheduled re-request of the disclosure set at renewal, with a named owner

Frequently Asked Questions

Should we publish our own model card if we build AI features on top of a vendor's model?

Publishing something is increasingly the right answer, but publish about your system rather than restating the vendor's model card. Your customers and your enterprise buyers are asking what your feature does, what data it sees, when a human is in the loop, and what it is not meant to be used for — none of which the upstream card addresses. There is a second reason to write it: producing the document forces someone in your organization to state the intended use, the out-of-scope uses and the known limitations in writing, and that exercise routinely surfaces disagreements between product and legal that would otherwise appear later in a support ticket. Keep it version-dated and update it when behavior changes, because a stale published claim is worse than no published claim.

What if the vendor refuses to give anything beyond the public card?

Then you know your risk position and you can price it. Record which items in the disclosure set went unanswered, what your use case depends on, and who accepted the residual exposure with a review date. Some of the gap can be closed on your side: you can evaluate the system on your own held-out data whether or not the vendor helps, you can pin a version if the API supports it, and you can log inputs and outputs so that you have a record of behavior over time. Those three do not require any vendor cooperation and they cover the questions most likely to be asked later. The unanswerable ones are training data and system internals, which is precisely where a documented acceptance decision belongs.

Is an ISO or management-system certification a better artifact than a model card?

It answers a different question. A management-system certification tells you the vendor has a governance process — defined roles, risk assessment, change management, monitoring — examined against a standard. That is genuinely more assurance than a self-published card carries, and it is about process rather than product. It does not tell you how the specific model behaves on your population, whether your use case is in scope, or what version is serving your traffic. The mature vendor file has both: a certification for process assurance, and use-case-specific evaluation evidence for product behavior. Neither substitutes for the other, and a review that stops at the certificate has answered the process question twice and the behavior question zero times.

How often should we refresh the documentation we hold?

Tie the refresh to events rather than to a calendar where you can: any model version change, any material behavioral change, any expansion of the use case, and any regulatory assessment you are producing. Add an annual floor at renewal, because renewal is when you have leverage and when someone is already opening the file. The practical failure is not the refresh interval — it is that nobody is assigned. Documentation aging is silent, produces no alerts, and only becomes visible when an outside party asks for the current version. A named owner with a calendar entry catches more of this than any policy sentence about maintaining up-to-date records.

Our vendor's card describes a base model from a different company. Whose documentation do we need?

Both, and the chain between them. The application vendor owes you documentation of their system — what they wrapped around the model, what they send upstream, what they filter. The base-model developer's card describes the component underneath. What you need in addition is the linkage: which base model and version the application currently uses, whether that can change without notice to you, and whether the application vendor's commitments about training, retention and deletion actually flow up to the model provider. That last one is where these arrangements break, because a well-drafted no-training promise from an application vendor is only as good as the terms they have with the provider behind them.

Ask Which Version Is Serving Your Traffic

It is one question, it takes an email, and the answer determines whether every other document in your vendor file describes the system you are actually running. If the endpoint resolves to a moving alias, the documentation you hold is about the past.

Pin the version where you can, get notice where you cannot, and date every artifact you file. Governance that cannot name the version is describing something else.