An accessibility exception that measures itself

Point an automated accessibility checker at a real product and you get a number. Fix everything it found and you get a smaller number. Somewhere in that process you will hit a finding that is genuinely not a defect, and at that moment you write your first exception.

That is the interesting part, because an exception list is where an accessibility programme quietly dies. It starts as two well argued entries. A year later it has eleven, three of which match nothing, one of which hides a real problem, and nobody remembers why any of them are there. The report says zero and means nothing.

We built the checker for our panel this month. It reports no findings at any severity, and this piece is mostly about why we do not believe that number on its own.

Three rules were off by default, and two of them were hiding findings

The library we use ships some rules disabled by default. We checked which ones, by asking it rather than assuming: name and content mismatch, table cells without headers, and target size are all off out of the box.

Two of those produced real findings on our panel, 36 between them, including both of the two points an earlier round had left open. A checker running the default rule set is therefore green on exactly the failures somebody had already gone looking for, which is the worst kind of green: it is indistinguishable from a product that is fine.

They are switched on explicitly, and that single decision changed the baseline more than any other configuration choice.

Nine languages, because one language is a coin flip

The language switcher in our header shows a two letter code and has an accessible name that says which language it is. In English the button reads EN and is called "Language: English", and the rule passes, because en happens to appear inside English.

In Dutch it fails. Same component, same code, same markup.

Measured across all nine languages we ship: three fail and six pass, and every one of those six passes by coincidence: en inside English, de inside Deutsch, fr inside Français, it inside Italiano, es inside Español, ro inside Română. Testing one language here is vacuously green.

So the marketing journey runs every language on the rules whose outcome can depend on language. It costs time and it is the only way that finding exists.

The baseline, and what it was made of

The first full run over marketing, sign in and the logged in panel reported 3 critical, 51 serious, 72 moderate and 6 minor. Eight rules accounted for all of it.

Almost all of it was structural rather than visual. The sign in pages and the panel had no main element and no skip link. The article tables had empty headers. Code blocks scroll horizontally and could not be reached by keyboard, so a keyboard user simply cannot read the right hand side of a long command. The statistics tab put a chart heading at the wrong level under the page heading.

After the repairs: zero, zero, zero, zero. Which brings us back to the question of what that zero is worth.

Two exceptions, each with a measurement

The exception list has exactly two entries, and each of them carries a reason and a measurement that was actually taken.

Colour contrast under forced colours. Running the panel with forced colours active produced 610 contrast failures across twelve pages. That is not a product defect, it is an interaction between the checker and the browser. The checker reads -webkit-text-fill-color before color, and Chrome does not force the former. On the sign in page, on the same heading, in the same page load:

So it compares the authored foreground against the forced background and reports 1.12 to 1 where the screen actually shows 21 to 1. In that mode the contrast requirement does not apply anyway, because the palette belongs to the user.

The rule is off in that one mode, and the reason is re-measured on every run. If Chrome or the checker fixes this, that self check fails and the exception has to be argued again. An exception without a self check is a silent allowlist.

The account button. The name and content mismatch rule fires on the user menu button in the panel. Six candidate accessible names were put on the real button and the rule was run against each. It passes only when the name literally contains the string DKDemo Klant€0.00, in which the initials and the account name have been concatenated, and it fails with that same text spaced out as DK Demo Klant €0.00, and with the real name. All three children of that button are hidden from assistive technology, which the rule ignores.

Passing would therefore mean a screen reader announcing DKDemo as one word. That is a worse button, adopted to satisfy a rule the library itself marks experimental and ships disabled. The functional requirement is met: someone using speech control says "Demo Klant", and that is in the name.

The exception that died without saying anything

This is the part worth the whole article.

An exception has to recognise the element it applies to. The checker's own CSS selector is not stable enough: the same account button was identified one way on one page and another way on the next, so an exception matching the selector covers one page and not the other. Instead the exception matches a marker attribute the component itself sets, in the HTML fragment of the element.

The fragment the checker reports is truncated at 140 characters.

The button's accessible label grew, because the account credit was added to it. The marker slid past the 140 character boundary. The fragment now ended ... Credit €0.00." data-us. The exception matched nothing, silently, and the run reported 27 serious findings it had covered the run before.

Nothing about that failure is loud. The exception did not error, the list did not shrink, no configuration changed. A change in an unrelated feature moved a string, and a piece of safety infrastructure stopped working. The fix is to match the untruncated opening tag, which the checker also provides.

The general lesson is bigger than accessibility: anything that matches against truncated text is a guard that fails at precisely the moment something changes, which is the only moment it was needed.

An exception that matches nothing makes the run fail

Every entry is counted. If a full run finishes with an exception that was hit zero times, the run is red.

That is the same shape as a list of constants elsewhere in this codebase, and for the same reason: an exception that no longer covers anything should be deleted, not left standing. Left standing, it is one more line that the next person has to reason about, and eventually it stops being read at all.

The counting only applies to a full run. Restrict the run to one journey or one mode and "not hit" means nothing.

Proving a zero is a real zero

Target size, the requirement that a control is at least 24 by 24 pixels, is one of the three rules the library ships disabled. We switched it on and wrote a second measurement of our own next to it, because the two exceptions the standard itself grants have to be measured rather than assumed. The final run examined 2056 targets. 155 were under the size. 70 of those fell under the standard's inline exception, meaning the target sits inside a sentence, and 85 under the spacing exception, meaning a 24 pixel circle around it touches nothing else. Zero remained.

A zero from a probe you wrote yourself is not distinguishable from a probe that cannot report anything. So a separate script injects a 16 by 16 pixel button next to a neighbour on a live page and runs this same probe text, read from the source file rather than copied, over it. It goes from 0 to 1 and names the element. The reporting path works, so the zero is a result rather than silence.

Both of the standard's exceptions are measured rather than guessed. Inline means the computed display value is exactly inline and the nearest block ancestor also contains other text, which is what "sits in a sentence" actually means. That is not a list of class names. And the spacing test measures the circle against the other targets' rectangles rather than their circles, which is stricter than the standard: it can produce a false red, never a false green.

What a machine cannot do

There is a short runbook of four gates that need a person, because they are the things no checker reaches: whether the reading order makes sense and an announcement actually arrives, whether a journey is doable with only a keyboard rather than merely reachable, whether the layout stays readable at 200 percent rather than merely fitting, and whether the screen still means anything in the user's own palette.

Those four are open. Saying so is part of the point: a report that says zero and does not say what it did not look at is the same failure as an exception without a measurement.

What we would write down

1. 01Ask the library which rules are off, do not assume. The disabled ones are where our findings were.
2. 02Run every language you ship on any rule whose outcome can depend on text.
3. 03Give every exception a reason and a measurement, written down where the exception lives.
4. 04Make an exception that matches nothing fail the run.
5. 05Never match a guard against truncated text.
6. 06Prove your zeroes by injecting something that should be caught.
7. 07Publish what a machine cannot check, so the number is read for what it is.

The same instinct, applied to translations rather than to markup, is in sixteen thousand keys in nine languages.

From the panel work on Provibr.