Three faults no script could see

We publish in nine languages. English is the source and the other eight are translations, and there is a wall of automation around them: a structure comparison that checks headings and tables and lists line up, a facts comparison that pulls the code fragments and the numbers out of every paragraph and demands they match, a typography test per language, a link check, a table check, a layout sweep at 390 pixels.

Eight articles had passed all of it. Then a second reader went through those eight in eight languages and found things to improve in every one: 44 in Dutch, 40 in German, 47 in Spanish, 35 in French, 39 in Italian, 22 in Portuguese, 55 in Romanian, 23 in Ukrainian, and 18 that crossed languages. The diff was exactly 307 lines out and 307 lines in across 62 files.

Most of those are ordinary translation quality: a register that slipped, a clause that reads as machine output, a term that is not what the industry says in that language. Three of them are a different kind of thing, and those three are the reason this is worth writing down.

A number that nobody measured

One article described a calibration probe: a script that plants deliberate defects on a page and then requires that the detector finds them, so that a run reporting zero findings can be distinguished from a run that was not looking.

Four of the eight translations said the probe requires three findings.

Nobody had checked, because a number in a sentence about somebody else's tooling looks like a fact you inherited. The script plants three elements and requires four kinds of finding. The source text, in English, said two. A note written during the review said three. Counting the translations, that is three statements of the number and two distinct values, and neither value is the one the script requires; each had been faithfully carried into several languages.

The repair was not to correct the number. It was to remove it. The sentence now names the three planted defects by what they are, and no translation states a count at all, because a count in that position adds nothing a reader needs and is a hostage to a script somebody will edit.

The general shape: a number that came from a summary rather than from the thing it describes will drift, and it drifts silently, because it is still a plausible number.

A sentence that lost its second half

An English sentence made a contrast: this is a refusal, not a fall back. In two languages only the first half survived. What remained was grammatical, fluent, and said something weaker and slightly different from what the source said.

The interesting part is how it was found. Six of the eight readers, working independently on their own language, flagged the same sentence. In six languages it was already right. Two had lost the contrast and were the reason it kept coming up.

A structure comparison cannot see this, because a paragraph that is one clause shorter is still a paragraph. A facts comparison cannot see it, because there is no number and no code fragment in it. The only thing that catches a dropped contrast is somebody reading the sentence and asking what it is contrasted with.

A title that disagreed with every link pointing at it

Our articles link to each other. The English source of article four had a title describing one thing sitting next to another, and every incoming link used the same wording.

In four languages the title had been translated as under and every incoming link as next to. So a reader clicked a link promising one thing and landed on a page whose heading claimed something else.

The link check passes this without complaint, and it is worth being precise about why: it verifies that the target of a link exists and responds. A link with the wrong text points at a page that exists. There is nothing for an HTTP check to notice.

The fix is a rule rather than a repair: the title of an article and every link that points at it are one string in a translation, and they get reviewed as one string. That is easy to state and easy to forget, because the two live in different files.

Two new guards, and why one of them has to count backwards

Two of these three are now automated, and the automation was proven red before it was trusted, on a copy of the tree with defects planted in it.

The first guard covers untranslatable names outside code fragments. The facts comparison already covers anything in backticks. What it cannot see is a product name, a protocol name or a menu item that stands in running text without them: those are exactly the tokens that translators translate, correctly by instinct and wrongly in fact. In one earlier round three languages had translated the name of a menu; in another, all eight had taken a CSS property out of a code fragment and turned it into prose.

The second guard covers numbers written as words. The facts comparison normalises digits across languages, so a thousands separator written as a full stop, a space or a comma all compare equal. It is by construction blind to a number spelled out, and in this house style almost every number in the English source is spelled out.

That guard has a subtlety that cost a first version its credibility. It counts occurrences of each value per article, in the source as a word or a digit and in each translation as that language's numeral or a digit. The naive implementation counts each value independently, and then "forty one" is also counted as an occurrence of "forty". On the first run that produced 37 false hits, which is more than enough for somebody to conclude the guard is noise and stop reading it.

The fix is to count from the largest value down and delete each match from the text as it is found, so that by the time "forty" is looked for, every "forty one" is gone. That is a small implementation detail with a large consequence: a guard that cries wolf is worse than no guard, because it trains people to skip a line in the output.

What the reader got wrong

One finding in the review was itself incorrect, and it is instructive.

The Dutch reader reported that emphasis on a particular word had been lost in all eight languages. Six of the eight had it. The Dutch reader could see one file properly and had inferred the rest.

That is now a rule: a reader working on one language reports on that language. A claim about eight files is a measurement, and it needs a script or eight open files, not an impression.

There was a related finding that turned out to be a genuine split rather than an error. Four of the eight Dutch articles used the formal register and four used the informal one. Across the whole Dutch catalogue, 47 of 90 files are formal. That is not a defect to fix in eight files; it is a decision that has to be made once and applied everywhere, and the round wrote it down instead of quietly picking a side.

Six things wrong in the source

The review also produced six findings that were not translation problems at all. The English was wrong, and eight languages had faithfully translated the wrong thing.

Those were deliberately not fixed in eight languages. They were reported against the source and handled in a separate round that changed the English first and then pulled each corrected sentence through the eight translations, so that the eight stayed translations of something rather than eight independent repairs of the same sentence.

That round found something too: the note describing one of the six was itself wrong about the numbers, which is where the count in the first section of this article comes from. A correction that is not verified against the source is just another claim.

What we would write down

1. 01Automation checks form and facts; it does not check meaning. Every one of these eight articles was green.
2. 02A number you inherited from a summary will be wrong eventually. Check it against the thing, or remove it.
3. 03The best repair for a fragile number is often no number. Name the things instead of counting them.
4. 04A dropped contrast leaves a grammatical sentence. Only a reader finds it.
5. 05Six readers flagging the same sentence is a signal about the source, not about six translations.
6. 06A link with the wrong text points at a page that exists. No HTTP check will ever notice.
7. 07A title and every link to it are one string. Review them together even though they live apart.
8. 08Prove a new guard red before you believe it green. Plant the defect on a copy of the tree.
9. 09A guard with false positives gets ignored, which is worse than not having it. Count from large to small and consume what you match.
10. 10A reader reports on their own language. A claim about eight files is a measurement.

For clarity about what this is: the facts comparison ran over 2432 paragraphs with no findings, the structure comparison over eight articles in eight languages with none, the token guard 248 checks and the numeral guard 240 checks with none, the content check 815 of 815 generated files byte identical, the web suite 4572 green, the metadata check 288 of 288, links 153 of 153, tables 198 of 198 and the layout sweep 168 of 168 at 390 pixels against a production build. The typography test was proven red at 28 pass and 3 fail with planted defects and green at 31 afterwards. Not proven: there is still one reader per language and none of them is a native speaker, and there was no accessibility or layout shift measurement in this round. Two terminology questions were escalated rather than decided.

From the platform work on Provibr. What nine catalogues do to a set of fixed controls is in sixteen thousand keys and nine languages, and a transformation script that damaged three articles it was never pointed at is in a conversion script that ran too wide.