Documentation written before the merge

A documentation page is a set of claims about code. That sounds obvious until you notice that nothing in the build checks any of them. Tests assert things about code. Type checks assert things about code. Prose about code asserts things about code and is verified by a person reading it once, on the day it was written.

We wrote two new documentation pages in one round: one for the automation editor, around 2930 English words across eight sections, and one for the dashboard, around 909. Plus a section added to two pages that already existed. Two sentences on the automation page were already false when it shipped.

Nobody had written a falsehood. Both sentences were accurate against the branch they were written on.

The sentence that was true when it was written

Documentation for a feature is usually written after the feature, which means it is written while other branches are still open. Ours was. The page described a notify node and said, in effect, that it does not send email. On the branch it was written against, that was exactly right: the node wrote a panel notification and nothing left the building.

A parallel round merged first and gave the node an email checkbox, off by default, an unsubscribe category of its own, and a rule that a test run never sends anything. Every one of those facts is interesting to a reader. None of them was on the page, and what was on the page now told them the opposite.

The second sentence described how a plan connects to a flow. It said the plan runs the flow behind it, from a time when that relationship was one-to-one. The same merge replaced it with a join: a plan may hang at most one flow per moment, out of three moments, and a manually triggered or webhook-triggered flow cannot hang there at all. "Cannot be attached" is a better answer than "attached and silently does nothing", but it is a different answer, and the page still gave the old one.

The repair was small. The interesting part is how it was found, which is that somebody went looking on purpose, because a round that touches a subject is the only moment anybody has a reason to reread the prose about that subject.

What a guard can catch

There is a second failure mode on the same pages, and unlike the first one it is mechanisable.

The dashboard page has a table listing the cards a user can add. Those cards have names, in nine languages, and those names already exist: they are the labels in the picker the user actually clicks. Writing a documentation table means writing them a second time.

It went wrong immediately, on the first draft, in English:

None of those is wrong as English. All three are wrong as documentation, because a reader who is told to add "Connections" will look for a card called Connections and find a card called Integration list. The drift is silent by construction: both texts are fine on their own, and nobody puts a manual next to a picker.

So the table now takes the same catalogue keys the picker takes, and a test requires them to be word for word identical in all nine languages. The documentation key is derived from the card id rather than listed by hand, so that the seventh optional card somebody adds next year is measured without anybody remembering to add it here. This is the same shape as the guard we run between the two copies of the English server messages, and it exists for the same reason: two places holding one truth need a test that says so, not a convention.

The button names on the same page were guessed in five of the nine languages. They now come out of the catalogue too. And three languages had translated the word Scripts, which is the name of a menu item that is not translated in the product, so the page told a Spanish, Portuguese and Ukrainian reader to click something that does not exist under that name.

A technical token is not a word

Product names, ports, paths, module names and status codes go in a monospace token and stay in English, wherever they appear, in every language. That rule is not new. What was new is that writing it into these pages collided with two other guards at once.

Putting a <code> element literally inside a table row array trips the lint rule that wants a key on every JSX element in an array, even here, where the table component wraps each cell in its own keyed element and the key is never read. Putting a key on it trips a different guard, which sees a keyed child sitting next to translation calls whose keys it cannot resolve in that file.

The way out was a one-line function that returns the element. To the lint rule it is a function call rather than an array element; to the other guard it is not a key. The rendered output is byte identical. It is a small thing, and it is the kind of small thing that gets solved with a disable comment when nobody writes down why the two rules disagreed.

Screenshots that looked successful

The measurement for these pages included anchor screenshots: scroll to each section, photograph it, look at it.

Nine of them showed the page header.

The browser command that captures a region beyond the viewport computes its clip from the top of the document, not from the current scroll position. Scroll to an anchor eight screens down, ask for a region at the coordinates the element reports, and you photograph the top of the page instead, at exactly the right size, in a file with the right name. Every one of those screenshots was a plausible looking image of the wrong thing.

Adding the current scroll offset fixed it. The general shape is one we keep meeting: a measurement that fails by producing normal looking output is worse than one that crashes, because the crash gets investigated on the same day.

The part no guard reaches

The card names were mechanisable because both sides are strings in one repository. The two stale sentences were not, and it is worth being precise about why, because it is tempting to believe every documentation problem has a test waiting to be written.

A sentence saying a node does not send email is a claim about behaviour spread over four files, a default value, a checkbox and a category list. There is no expression you can compare it to. The only mechanism that would catch it is a test that renders the behaviour and reads the sentence, which is a test that has to be written per sentence, by somebody who already knows the sentence is wrong.

What we did instead was cheaper and worth naming as a practice: when the correction was made, the whole documentation subtree was searched for any other claim about email and any other claim about a flow per plan. There was no third place. That is a result, not a formality. Without it, the round would have fixed two sentences and left the reader with the reasonable belief that the page had been checked.

The same discipline already exists elsewhere in this project for a different pair: our long form articles can go stale against the list of which integrations are live, and nothing turns red when they do. Two articles once spent six weeks saying two hypervisors were on the roadmap while both had been shipping. The rule that came out of that is the same one: if a round changes the status of a thing, grep the prose for its name.

What we would write down

1. 01Prose about code is an untested assertion. Nothing in the build reads it. That is fine, as long as everybody knows it and nobody assumes a green run says anything about the paragraph.
2. 02Documentation is written against a branch, not against main. If other rounds are open, the page is a snapshot of a state that may never ship. Reread it after the merge, not before.
3. 03A name that exists in the product should be read, not retyped. Three of six card names drifted on the first draft. In nine languages that is a class of error, not an accident.
4. 04Derive the documentation key from the id. Then the next card is covered without anybody remembering this rule.
5. 05A guessed word in a language you do not read is invisible. Five languages had guessed button names and nothing was red.
6. 06Translating a product name is a broken instruction. If the menu says Scripts in every language, the manual says Scripts in every language.
7. 07When two lint rules disagree, write down which one you obeyed and why. The alternative is a disable comment nobody can evaluate later.
8. 08A measurement that produces plausible output when it fails will not be questioned. Photograph something you can recognise, and look at the pictures.

For clarity about what this is: both pages are live in nine languages, the two stale sentences were corrected in all nine, and the card name guard is in the suite and was proven red before it was proven green. The claims on the pages come from reading the code rather than running it; the curl example on the automation page has not been fired at a live endpoint; and each translation has had one reader, not two.

From the platform work on Provibr. The cards those names refer to are in a shortcut is a reference, not a right, and what nine catalogues cost to keep honest is in sixteen thousand keys and nine languages.