What is the single most useful test to run during an accessibility tool trial?

Plant defects you already understand and see what comes back. On a staging copy of a real template, introduce a short list of known failures — an image with no alternative text, an input with no programmatic label, a div acting as a button with no role or keyboard handler, a contrast ratio just under the threshold, a heading level skipped, a modal that does not trap focus, an iframe with no title. Then read the report against your list. You learn three things at once that no demo shows you: what the tool detects, what it misses silently, and how it describes a defect whose correct answer you already know. A tool that reports your seeded contrast failure with the exact ratio and selector is telling you something very different from one that reports a generic colour warning on the page.

How do I measure a scanner's false-positive rate in two weeks?

Take a fixed sample rather than the whole report — fifty findings, chosen across severities and rules, not the first fifty. Have someone check each one by hand against the success criterion it cites and record three outcomes: genuinely failing, technically firing but not a real barrier, and wrong. The proportion in the second and third buckets is the number that decides whether your developers keep using the tool after month two. It is worth doing this the same way on every vendor in the evaluation, because the rate varies enormously by how aggressively a vendor tunes for recall, and a platform that surfaces more issues is not necessarily finding more problems.

Should the trial run against production or staging?

Both, for different reasons. Staging is where you seed defects and control the variables, because you cannot plant a broken button in production. Production is where you find out whether the crawler survives your CDN, your bot protection, your rate limits and your consent banner — all of which routinely defeat scanners and none of which appear in a demo against the vendor's sample site. Run the seeded test on staging and a full unaided crawl on production, and compare the crawled-URL count to your own sitemap to see how much the tool actually reached.

What should I ask vendor support during the trial?

Pick one genuinely ambiguous finding from your own report and ask them to explain why it fires and what the correct fix is. This tests the thing you will actually depend on, which is not uptime but whether a human on their side understands WCAG well enough to adjudicate a disputed result. Note how long the reply takes, whether it cites the specific success criterion, and whether it engages with your markup or returns boilerplate. A trial is also the period of maximum attention you will ever receive from a vendor, so a slow or generic answer now is the optimistic case, not the pessimistic one.

How long should an accessibility platform trial be?

Long enough to contain at least two scheduled scans plus one deliberate change in between, which for most platforms means two weeks and for a monthly cycle means asking for an extension. The reason is that half of what you are buying is the second scan: whether a fixed issue closes cleanly, whether a newly introduced regression is flagged and how quickly, and whether the diff between two scans is readable. A single-scan trial only evaluates the tool as an auditor, and you are buying it as a monitor.

Is a free scan enough, or do I need a full trial?

A free scan is a good first filter and a poor decision basis. It tells you whether the tool can reach your site at all, roughly what its rule set catches on a public page, and how it presents a finding — enough to cut a shortlist from six to three at zero cost. What it cannot tell you is anything about authenticated coverage, regression alerting, export shape, false-positive rate on your own codebase or support quality, which are the four things that determine whether the subscription survives a year. Use free scans to shorten the list, then run one structured trial on the finalists rather than three shallow ones.

The Two-Week Accessibility Vendor Trial: What to Actually Test (2026)

Test One: Seed Defects You Already Understand

The problem with reading a scan report of your own site is that you do not know the right answer, so you cannot tell detection from noise. Fix that by controlling the input. On a staging copy of a real template — not a blank test page, because context affects several rules — introduce a short list of failures whose correct verdict you know in advance.

An informative image with no alternative text, and a decorative one with a redundant description.
A text input whose only label is adjacent placeholder text.
A div styled as a button with no role, no tabindex and no key handler.
A body-text colour pair just under the 4.5:1 threshold, and one just over it.
A heading level skipped from h2 to h4.
A modal that opens without moving focus and does not trap it.
An iframe with no title, and a link whose only text is "read more".
A form that shows an error visually without announcing it programmatically.

Then read the report against the list. Roughly the first half of that list is reliably machine-detectable and any serious tool should return all of it; the modal focus behaviour and the unannounced error are the interesting ones, because they are where automated coverage genuinely ends and where a vendor's marketing is most likely to have overpromised. What you are grading is not the count but three specific things: what came back, what was silently missed, and whether each finding names the element and the success criterion precisely enough to act on without re-investigating.

Keep the seeded page and reuse it on every vendor in the evaluation. It is the only part of the process that produces a directly comparable result across platforms.

Test Two: Point It at the Pages That Break Crawlers

Most accessibility failures that generate complaints live behind a login, inside a multi-step flow, or in a component that only renders after an interaction — and most of the parts of a site that a scanner cannot reach are exactly those. A tool that cannot authenticate is measuring your marketing pages and reporting the result as though it were your product.

During the trial, deliberately aim it at the hard surfaces: the authenticated dashboard, a checkout or application flow past step one, a route that only exists client-side, a page behind your consent banner, a page your bot protection challenges, and a representative PDF. Then compare the tool's crawled-URL list against your own sitemap. The gap is the part of your site the subscription does not cover, and it is almost never mentioned unless you ask.

Ask specifically how authentication is configured — stored credentials, a session cookie, a recorded login script — and whether it survives a password rotation or an MFA prompt, because a login integration that breaks silently every ninety days turns into a monitoring gap nobody notices.

Test Three: Count the False Positives Yourself

The number that decides whether a platform is still in use in month six is not how much it finds, it is how much of what it finds is real. Developers abandon accessibility tooling for one reason above all others: they fixed three things that turned out not to be broken and stopped trusting the queue.

Measure it rather than guessing. Take fifty findings sampled across rules and severities — not the first fifty, which are usually the same rule repeated — and have someone competent check each against the success criterion it cites. Record three outcomes: a genuine barrier, a technically-correct firing that blocks nobody, and a plain error. Run the identical exercise on each finalist. A platform reporting four hundred issues at a thirty percent junk rate is worse than one reporting a hundred and twenty clean ones, and the dashboards will tell you the opposite.

Test Four: Turn One Finding Into a Ticket

Export a single finding and try to hand it to a developer who has not seen the dashboard. That is the transaction the subscription exists to perform, repeated a few hundred times a year, and it either works or the tool becomes a report nobody opens.

A usable finding carries the URL, a stable element selector, the specific success criterion, the observed and expected behaviour, and enough context to reproduce it. A screenshot is a nice addition and not a substitute for a selector. Check the export format is structured — CSV, JSON, or a direct integration that creates the issue with those fields populated — rather than a PDF someone has to retype. And check the identifier is stable across scans, because without that, the retest cannot say whether this is the same issue or a new one, and your trend line is fiction.

Test Five: Break Something on Purpose

Halfway through the trial, fix one seeded defect and introduce a new one. This is the only test that evaluates the product as a monitor rather than as an auditor, and it is the half of the purchase most evaluations never exercise.

Three things to watch. Does the fixed issue close automatically, or does it linger as open and require manual housekeeping that will not happen. Does the new defect get flagged, and how long after deployment — an alert that arrives with the monthly report is not monitoring. And is the diff between two scans presented at all, or do you get two full reports and the job of comparing them yourself. The answer to the third question is what determines whether anyone reads the second month.

Test Six: Ask Support Something Hard

Find a finding in your own report that you genuinely cannot adjudicate and send it to support with your markup. Ask why it fires and what the correct remediation is. You are testing whether there is accessibility expertise behind the product or only a tier-one queue, and that distinction matters every time a developer disputes a result — which, over a year, is often.

Record the response time, whether the answer cites the specific success criterion, and whether it engages with your code or restates the rule description you already read. Remember the framing: a trial is the period of maximum vendor attention you will ever have. Whatever you get now is the ceiling.

A Two-Week Schedule That Fits Around Real Work

Day 1: seed the staging template; start the unaided production crawl.
Day 2: compare crawled URLs against your sitemap; configure authentication.
Day 3: read the seeded report against your list; note misses.
Day 4: sample fifty findings and hand-check them.
Day 5: export one finding; hand it to a developer cold.
Day 6: send support the hardest ambiguous finding you have.
Day 7: fix one seeded defect, introduce one new one.
Days 8-14: wait for the second scan; grade closure, detection latency and the diff.

Questions That Come Up During Evaluations

The vendor wants to run the trial for us. Should we let them?

Let them configure it and then run it yourself. A vendor-driven trial optimises for a good first report: they will scope the crawl to pages that scan well, present the findings in the order that reads best, and quietly avoid the surfaces their crawler struggles with. None of that is dishonest, and none of it tells you what year two looks like. Accept the onboarding help, then insist that the seeded test, the authenticated crawl and the fifty-finding sample are done by your team on your inputs.

How many vendors should we trial at once?

Two, occasionally three. Each structured trial costs roughly a day of someone's real attention spread over two weeks, and the exercise only produces comparable results if the same person does it the same way on each. Use free scans and published documentation to cut a longer list down first, on the structural questions — does it authenticate, does it export structured findings, does its pricing unit match how our site grows — and reserve the trial for finalists that pass all three.

Our developers say the tool's findings are wrong. How do we settle it?

Settle it on a specific element against the specific success criterion, not on the aggregate. Pull the disputed finding, read the criterion text, and check the behaviour with a keyboard and a screen reader. Often both sides are partly right: the rule fires correctly on the markup while the actual user experience is fine because of something the scanner cannot see, which is a legitimate suppression with a written reason. That process is also the reason your evaluation should measure the false-positive rate up front — a tool that forces this conversation weekly will lose the team regardless of who is right.

Does the trial tell us anything about legal exposure?

It tells you what the automatable layer looks like, which is a minority of the WCAG success criteria and roughly the layer a complainant's own consultant will run first. That is genuinely useful as a floor. It does not establish conformance, and no scan-based report should be presented as though it did — the criteria that need a human are the ones that produce the experiences people complain about. Use the trial to size the machine-detectable backlog and to decide what is left for a manual audit.

What if the trial expires before we have finished?

Ask for an extension and expect to get one, particularly if you can point at a specific unfinished test. The request also functions as a signal: a vendor who will not extend a trial by a week during an active evaluation is telling you something about how the relationship runs once the invoice is paid. If an extension is refused, prioritise the second scan over everything else — the regression and closure behaviour is the part you cannot infer from documentation.

Should we test the same pages on each vendor?

Yes, and fix the list before the first trial starts. Comparability is the entire point, and it evaporates the moment one tool is pointed at your marketing site and another at your product. Write down the exact URL set — a few templates, one authenticated flow, one document, your seeded page — and use it identically on every candidate. The same discipline applies to the finding sample: same sample size, same reviewer, same three outcome buckets.

Start the Evaluation With a Reading You Own

Before the first sales call, take your own scan of the templates you intend to put in scope. It costs nothing, it gives you a dated reference point that belongs to no vendor, and it means you can tell instantly whether a trial report is finding something new or re-presenting what a free tool already returned.

Then grade each finalist on the six tests: seeded detection, authenticated coverage, false-positive rate, export shape, regression latency, and the quality of one support answer.