A conversion script that ran too wide

French and Italian require a typographic apostrophe rather than a straight one, and there is a test that enforces it. Writing that character directly into a source file is unreliable: depending on how the file is produced, a shell, a heredoc or an editing tool can flatten it back. So we write the straight character and run a small script that converts it afterwards.

That script masks code. Anything inside backticks and anything inside a fenced block is left alone, because an apostrophe inside a code fragment would change the fragment, and the facts comparison would correctly go red.

It was run as a glob over a whole language folder. It converted an apostrophe inside a quotation of program output, in an article written by a different round, that happened to be sitting in the same folder.

The masking was correct and the target was wrong

The line it damaged was a quotation of what a command printed, set inside guillemets as running prose rather than marked up as code. Something of the shape Locking administration user's password. That apostrophe belongs to the machine that emitted it, and curling it makes the article claim the machine said something it did not.

The masking rule is right. Backticks and fenced blocks are code. A quotation of program output that is typeset as prose is not marked as code and cannot be distinguished from prose by looking at the characters, because that is precisely what it is: prose, quoting a machine.

So the defect is not in the masking. It is in the selection. Three articles from other rounds were modified by a script that was written to touch four.

Why the build check did not notice

This is the part that generalises beyond one script.

Our articles are generated. The source is markdown, the output is committed generated files, and there is a check that rebuilds from source and demands the result is byte identical to what is committed. If those diverge, it goes red.

They did not diverge. The script changed the source, the build was run, the output was regenerated from the changed source, and the two matched perfectly. The check answers "is the output consistent with the source", which is a real question and not this one. Nothing in the pipeline answers "did this round change only what it was supposed to change".

Neither did the typography test, for the same reason in reverse: a curled apostrophe in French is the correct character. The change made the file more compliant, not less.

The only thing that showed it was git status on paths outside the round's own files. That is now the rule: after running any transformation, list the changed files and confirm every one of them is yours.

The rule that follows

Run a transformation script on the file names of your own round. Never on a glob over a directory.

That sounds obvious written down, and it was not obvious while writing it, because a glob over a language folder is the natural way to express "convert the French articles" and the script had a correct safety property that made it feel safe: it demands at least one replacement per file, so that a run which silently does nothing cannot report success. That check is about vacuous success. It says nothing about scope.

A safety check that protects against doing too little is not a safety check against doing too much. Those are separate properties and they need separate mechanisms.

Two smaller versions of the same problem

The round that hit this also hit two translation faults that come from the same family: something that should have been immune to editing was not marked as such.

A caching directive appeared in the source as bare prose rather than in backticks. Two languages translated it, which turns a keyword into a phrase, and one of them additionally turned a duration of one year into a plural. Neither is caught by the facts comparison, because that comparison covers what is inside backticks and what is a digit, and a bare keyword is neither. The repair is to mark it as a token in all nine languages, after which the comparison covers it permanently.

The second was a table of interface labels. The English article used it to show that two lists of names had drifted apart, and stated explicitly that the drift was measured in English. Six of the eight translations translated the table. A translated pair of names is not a milder version of the measurement; it is a different measurement, one that was never taken. That took 26 replacements to undo.

Both faults are the same shape as the apostrophe: a piece of text that is data rather than language, sitting in a position where nothing marks it as data.

What a check answers is narrower than what it seems to answer

The three mechanisms in play here are each correct and each answered a question adjacent to the one that mattered.

None of these is badly designed. The failure is the assumption that a wall of green checks constitutes coverage, when what it actually constitutes is coverage of the things somebody thought of.

The cheapest counter to that is not another check. It is looking at the diff, at the level of file names, before committing anything a script produced.

What we would write down

1. 01Scope a transformation to the files of your own round. A glob over a directory is not scope, it is a category.
2. 02A guard against doing too little says nothing about doing too much. They are different properties.
3. 03A quotation of program output is data. Mark it as code, or a text transformation will treat it as language.
4. 04A consistency check between source and output is blind to a change in both. It answers about agreement, not about intent.
5. 05A change that makes a file more correct is the hardest kind to notice, because every quality signal goes the right way.
6. 06Read the file list of your own diff. It is the only check that answers the scope question.
7. 07Anything that is a keyword belongs in backticks in the source, in every language, or somebody will translate it and no test will care.
8. 08A translated measurement is an invented measurement. If the point of a table is that two things differ, the table is evidence and not prose.

For clarity about what this is: the damage was reverted, the round finished with the content check at 815 of 815 generated files byte identical, the web suite at 4546 green, the package, gateway and tools suites at 7236 green, the facts comparison over 1240 paragraphs with no findings, links at 81 of 81, tables at 99 of 99 and the layout sweep at 108 of 108 at 390 pixels against a production build. The typography test was proven red with two planted defects and green afterwards. Not proven: there is one reader per translation, three languages were checked only on layout, typography and HTTP status, and the figures quoted from earlier rounds were not re-measured here.

From the platform work on Provibr. What a second reader found in the same eight articles is in three faults no script could see, and what nine catalogues do to a fixed set of controls is in sixteen thousand keys and nine languages.