Two accessibility tools gave us completely different scores. Which one is right?

Probably both, because they are not measuring the same quantity. An accessibility score is not defined by WCAG — the standard has no scoring system at all, only success criteria that are met or not met on a given page. Every score you have ever seen is a ratio a vendor invented: some denominator of checks, some counting rule for instances, some severity weighting, over some sample of your pages. Change any one of those four and the number changes without a single line of your markup changing. The useful question is not which score is right but which findings are real, and that is answered by reading individual findings, not by comparing headline figures.

Why does one tool report 40 issues and another 4,000 on the same page?

Almost always the instance-counting rule. If a shared navigation component has one unlabelled control and your site has two thousand pages, a tool counting per failing element across the crawl reports it four thousand times, a tool counting per page reports it two thousand times, and a tool that deduplicates by component and rule reports it once. All three are describing a single defect with a single fix. This is the biggest single source of apparent disagreement between reports and the easiest to check: pick the largest issue category in each report and ask how many distinct fixes it represents. Ask the same question of any vendor quoting remediation by issue count.

Do the scanners use different rule sets?

They overlap heavily and differ at the edges, and the edges matter. Most commercial platforms build on a small number of shared open-source rule engines, so the core detections are similar — but vendors pin different engine versions, enable different optional and experimental rules, add proprietary checks of their own, and make different calls on rules with known false-positive rates. Two tools running the same underlying engine a few versions apart will disagree about specific findings. When comparing reports, ask which engine and version is running and whether experimental rules are on, because that alone explains a visible slice of the gap.

Could one scanner simply have seen less of the site?

This is the most common explanation for a suspiciously good report and the one to check first. Crawl depth, page limits per tier, robots handling, bot protection, consent banners, authentication, client-side routing and script timeouts all change what a scanner reaches. A tool that crawled two hundred marketing pages and a tool that crawled two hundred pages including your authenticated dashboard are not measuring the same site. Before comparing anything else, compare the crawled-URL lists against each other and against your sitemap. A cleaner report from a smaller sample looks identical on a dashboard to a genuinely cleaner site.

Does automated testing cover most of WCAG?

No, and this is the reason two reports can agree perfectly and still both be incomplete. Automated rules reliably decide a minority of the success criteria — the ones with a machine-checkable answer, such as missing alternative text, unlabelled controls, contrast ratios, document language, heading order and clearly invalid ARIA. Whether alternative text is accurate, whether focus order makes sense, whether an error is announced, whether a custom component behaves like the control it imitates, and whether a flow can be completed with a screen reader are all human judgements. Each vendor also chooses a different subset of the automatable layer to emphasise, which is why their scores diverge even though their blind spots are similar.

How do we make two vendor reports actually comparable?

Fix the inputs and ignore the headline. Give every vendor the same explicit URL list rather than letting each crawl freely. Ask each to report unique defects and distinct fixes alongside raw instance counts. Ask which rule engine and version ran and whether experimental rules were enabled. Then hand-check the same sample of findings from each report against the success criterion cited, and count genuine barriers rather than rows. What comes out of that is a comparison of detection quality on an identical sample, which is a decision basis. A pair of scores is not.

Why Two Accessibility Vendors Score the Same Site Differently (2026)

There Is No Such Thing as a WCAG Score

It is worth starting here because it explains everything downstream. WCAG is a list of success criteria, each of which is either met or not met for a given page at a given conformance level. It defines no percentage, no grade and no index. Every score, letter grade and compliance percentage you have seen on an accessibility dashboard was designed by the company showing it to you, as a way of compressing a long list of pass/fail results into something a manager can watch move.

That is not inherently dishonest — a single trackable number has real operational value — but it does mean a score is only comparable against itself, from the same tool, on the same crawl configuration, over time. The moment you set two vendors' scores side by side you are comparing two different formulas applied to two different samples, and the comparison carries no information at all.

Difference One: What Was Actually Scanned

Check this before anything else, because it explains more discrepancies than the other five combined and it is the only one that can make a worse site look better. Tools differ in crawl depth, per-tier page limits, how they treat robots directives, whether they can authenticate, whether they render client-side routes, how long they wait for scripts, and whether your bot protection or consent banner stops them at the door. A scanner that never got past your cookie wall will report a very clean single page.

Pull the crawled-URL list from each report and compare it against your sitemap and against the other tool. Look specifically for the authenticated area, the multi-step flows, the client-side routes and the document library — those are where the defects that generate complaints actually live, and they are the first things a crawler misses.

Difference Two: How an Instance Is Counted

This is the source of the spectacular gaps — the forty-versus-four-thousand cases. One unlabelled control in a shared header is, depending on the counting rule, one defect with one fix, or one finding per page across the whole crawl, or one finding per failing element on every page. All three reports describe exactly the same work.

The practical test is quick: take the largest issue category in each report and ask how many distinct code changes would clear it. In component-based front ends the answer is frequently one. This matters commercially as well as diagnostically, because remediation is sometimes quoted per issue — and a quote built on an element-level count of a templated defect is pricing one fix several thousand times.

Difference Three: Rule Set and Engine Version

Most commercial scanners are built on a small number of shared open-source rule engines, which is why their core findings overlap so much. The divergence comes from the choices layered on top: which engine version is pinned, whether experimental or best-practice rules are enabled alongside the WCAG-mapped ones, which proprietary checks the vendor has added, and how aggressively they tune rules with known false-positive behaviour.

Ask both vendors which engine and version they run and whether non-WCAG best-practice rules are counted in the headline figure. A platform that folds best-practice advice into the same number as conformance failures will always look stricter, and the extra findings may be worth having — but they are not the same category of obligation, and they should be separable in the export.

Difference Four: Severity Weighting

Once findings are collected, each vendor collapses them into a grade using its own weights. Some weight by WCAG conformance level, some by an internal user-impact scale, some by frequency across the crawl, some by a blend. A site with one severe blocker in checkout and a long tail of minor contrast issues can score well under one weighting and badly under another, and the one that scores it badly is not necessarily the more useful report for your team.

If a score is going to be reported upward inside your organisation, find out what moves it before you adopt it. A score dominated by instance counts will reward bulk template fixes and stay flat when you repair the one flow that was genuinely unusable — which trains the team to do the wrong work.

Difference Five: When the Page Was Measured

A modern page is not a fixed artefact. What a scanner sees depends on when in the render lifecycle it snapshots the DOM, whether it waits for lazy-loaded content, whether it interacts with anything at all, and what the viewport size was. A menu that is accessible once opened is invisible to a tool that never opens it; a modal that fails focus management cannot be evaluated by a tool that never triggers it; a contrast failure in a hover state is only found by a tool that simulates hover.

Personalisation, A/B tests, consent-dependent content and region-based routing add genuine run-to-run variance on top of that. If two reports disagree about a specific page, re-run both against a stable staging copy before concluding anything about either tool.

Difference Six: What the Denominator Is

Even with identical findings, the ratio can differ. A score computed as passing checks over total checks run rewards a tool that runs many trivially-passing checks. A score computed over pages rewards a site with many simple pages. A score computed over WCAG success criteria applicable to the page is the most defensible and the least common, because it produces a number that barely moves and therefore makes a poor dashboard.

This is also why an accessibility score should never appear in an accessibility statement or a conformance claim. The statement should say what standard was applied, what scope was tested, when it was last reviewed and how to report a barrier. A vendor-specific percentage in a public document invites exactly the question you cannot answer: compared to what.

How to Compare Two Reports Honestly

Give both vendors the same explicit URL list instead of letting each crawl freely.
Compare crawled-URL counts first; resolve coverage before discussing findings.
Ask each for unique defects and distinct fixes alongside raw instance counts.
Ask which rule engine and version ran, and whether best-practice rules are in the headline number.
Hand-check the same sample of findings from each report against the criterion cited.
Count genuine barriers, not rows — then compare those two numbers, and nothing else.

Questions This Usually Raises

A vendor's free scan gave us a much worse score than our paid platform. Is that a sales tactic?

It can be, and it can equally be a methodology difference you can verify in ten minutes. Free public scans typically run one page with every rule enabled including best-practice ones, and count per element — a combination that maximises the number on screen. Your paid platform may be counting per page across a crawl with experimental rules off. Before assuming bad faith, check three things: how many URLs each looked at, whether the finding categories are the same, and how many distinct fixes the worst category represents. If the free scan names specific defects your platform never reported, that is a real detection gap and worth pursuing regardless of the score.

Our developers fixed a batch of issues and the score barely moved. Why?

Usually because the fixes cleared a small number of instances in a score dominated by a different rule, or because the crawl picked up new pages in the same cycle and the denominator grew. Check the issue-level diff rather than the headline: confirm the specific findings you targeted are now closed, and look at whether the crawled-URL count changed between the two scans. A score that does not respond to real fixes is a signal to track closed findings by identifier instead, and to stop reporting the score upward.

Should we run more than one scanner permanently?

Some teams do, and the honest case for it is coverage of the edges rather than a second opinion on the score. A second tool is worth its cost mainly where it reaches something the first cannot — a native mobile app, a document estate, a component library in CI — or where you need an independent reading for an external audience. Running two general-purpose site scanners side by side mostly produces two overlapping backlogs and an ongoing argument about which number to report, which is a governance cost with no detection benefit.

An external complaint cited a score we do not recognise. What now?

Ignore the score and work the findings. Ask, or work out, which tool produced it and against which URLs, then reproduce the specific barriers by hand with a keyboard and a screen reader. What matters in that conversation is whether a real person is blocked from a real task, which is a question about specific journeys rather than about an index. Fix and document the blocking issues with dates, and keep your own dated scan history as the record of what you knew and when.

Which number should we actually track internally?

Track open defects by severity from a single tool, plus time-to-close, plus the crawled-URL count next to both so a coverage change cannot be mistaken for progress. Those three move for reasons you can explain, and they survive a methodology change in a way a composite score does not. Keep the vendor score if people like watching it, but treat it as a dashboard ornament rather than as the metric the programme is managed by.

Does any of this mean automated scanning is not worth doing?

The opposite — it means the scan is worth doing for its findings rather than its grade. The machine-detectable layer is cheap, repeatable, catches real barriers, and is very often where a complainant's consultant starts. Run it continuously, act on the individual findings, and use a manual audit for the criteria that need a person. The failure mode is not automated testing; it is managing an accessibility programme against a composite number that six methodology choices can move without anything changing.

Get a Third Reading You Control

When two vendor reports disagree, the cheapest way out is a neutral reading of your own on the same URLs — one you can re-run whenever you like, with a crawl list you set. It will not settle the score question, because nothing does, but it will tell you which specific defects both reports should have contained.

Then compare the three on findings: same URLs, same sample, same hand-check, counting genuine barriers rather than rows.