A run that printed its own failure

Before a release we run one script. It executes the web test suite, the package and gateway and tools suites, ten type checks, the lint, the content check, the protocol lint and the protocol code generation, and it writes one line per step into a summary file: the name of the step and its exit code. Then it writes DONE.

It wrote DONE even when some of those exit codes were not zero.

On the seventh of September a release went out while the summary contained a line reading bun test packages/gateway/tools exit 1. I had read the pass and fail counters of two suites and not the third. Nothing broke: the failing test was a guard about the headers of measurement scripts, so the product was unaffected and the customer never saw anything. The outcome was fine and the shape was wrong, and the shape is the only part worth writing down.

The counters are not the gate

The reason this is embarrassing rather than merely unlucky is that the failure mode has a name in our own notes, and the name is vacuously green: a check that is not wired to a decision does not get read, and a check that is read by eye gets read the way a person reads, which is by looking for the thing they expect to see.

The summary was designed to be scanned. Each suite prints its own totals somewhere above, so the natural motion is to look at "4515 pass, 0 fail" and feel finished. The exit codes were sitting one screen further down in a file whose last line said DONE.

The repair is four words of shell. After all the steps, the script greps its own summary for any exit code between one and nine, and if it finds one it appends a line beginning with FAIL:. The rule that goes with it is that a DONE without a FAIL is the gate, and the counters are not. The second half of that rule matters more than the first: a person who has a cheap correct signal will stop building expensive habits around an unreliable one.

There is a companion rule, which is that a deploy command never shares a shell chain with a verification that only prints. If the verification cannot fail the chain, it is documentation.

What was actually red

The guard that failed is one of a family, and it is worth describing because it is unusually cheap for what it buys.

The gateway directory holds the measurement scripts: browser sweeps, layout probes, seeding and cleanup tools, the things a round writes to prove something and then leaves behind. There are just over five hundred of them today. They are plain scripts with no type check, which means the only thing standing between them and rot is that somebody happens to run one.

Four sieves run over all of them in a normal test suite:

1. 01Every script parses, using the runtime that actually runs it.
2. 02Every relative import resolves to a file that exists. That is the form of dead a parser cannot see: the script is syntactically perfect and falls over on the first line when you run it.
3. 03Every script carries a header with prose in it. Without an explanation, a tool from four rounds ago is indistinguishable from one that still matters.
4. 04Every header names the round it came from, in one of exactly two fixed forms.

The fourth one was red. Three scripts from the documentation round had headers, good ones, and no round marker. That is the whole incident: a naming convention, caught by a guard that exists so that five hundred scripts stay legible, sitting in a summary line nobody read.

Why unknown has to cost a commit hash

The fourth sieve accepts two shapes and nothing else. Either the header says the round name with a date and where that attribution came from, or it says the round is unknown and carries the commit hash of the file's first commit, with a date.

The name is explicitly not allowed to be the word unknown. Without that exclusion, "round unknown, 2026-09-04" is a perfectly valid name-with-a-date, and the expensive branch can be avoided with a word. As it stands, admitting you do not know costs you a hash that somebody has to go and look up, and by the time they have looked it up they usually know the round.

The date is mandatory for a different reason. A bare name is prose, not a marker: a sentence like "the round that also touched the header work" contains a round name and asserts nothing about the file it is in. With a date attached it becomes a claim about this script, made on a day, which is the smallest thing you can hold somebody to.

A gate has to be at zero

The obvious question about five hundred untyped scripts is why we do not type check them. We measured that, and the answer is instructive.

The configuration exists. It is an instrument, not a gate, and it does not run in the verification script. Measured across the scripts as they were:

A gate must sit at zero, and those remaining 159 are mostly not defects. Two families were walked and both turned out to be artefacts of typing untyped code: a parameter without a default that falls out of the inferred shape, and a destructuring loop over a mixed array that widens to a union. Twenty seven of them are one and the same diagnostic, about importing a TypeScript file from a script that runs under a loader which handles it perfectly.

What the instrument did buy is the 398 to 159 step. One line in a shared browser helper, a constructor assigning its arguments onto the instance in bulk, kept 239 errors alive, because every property access on that object in every script was unresolvable. One repair, 239 errors gone. That is a good return for a measurement that is allowed to be non-zero, and it is exactly the kind of return you do not get from a gate, because a gate that cannot be satisfied gets switched off.

The check that measured its own runtime

Getting the parse check to work took two attempts, both of which failed in ways that looked like product problems.

The first version asked the current runtime to check each file. Under our test runner the current runtime is not the runtime the scripts run under, and its check flag means something else entirely. The result was 301 reports of syntactically broken files, with source lines quoted as error text. Every one of them was fine.

The second version named the correct runtime explicitly and still failed, differently: the test runner reaps child processes belonging to a test, and from roughly halfway through the list every call came back with a null status. Killed by a signal, not a parse error. A guard that goes red at random is not a guard.

The version that works spawns one child process which itself checks the files, eight at a time, in about nine tenths of a second when that was measured. What is being asserted is unchanged: whether the runtime that runs these scripts will accept them.

The parallelism is not a nicety either. Run serially, the check exceeded the per-test time limit, and a guard that times out reads as a broken guard rather than a slow one.

The lower bound nobody asks for

There is a fifth test in that file which asserts only that there are more than fifty scripts to check.

A directory walker that finds nothing makes every assertion after it trivially true. Rename the directory, change the extension filter, break the walker, and four green sieves report on an empty list. The lower bound is deliberately far below the real number, because the point is not to prevent cleanup; it is to make the difference between "nothing is wrong" and "nothing was examined" a visible one.

The same test file is honest about its limits, and that honesty is part of the design. None of the four sieves proves that a script still measures what it claims. A selector that no longer exists, a route that moved, a column that was dropped: all four checks pass. It is a floor, not a type check, and saying so in the file stops the next person from treating a green run as more than it is.

What we would write down

1. 01A verification that only prints is not a gate. It has to be able to fail the thing that comes after it.
2. 02Write the failure line, not just the exit codes. The gate is a word somebody can grep for, at a fixed place, not a number they have to notice.
3. 03Never chain a deploy behind a check that cannot stop it. The chain is the promise.
4. 04Counters are a summary, not a verdict. People read the number they expect to see.
5. 05A gate must be able to sit at zero. If a measurement cannot, keep it as an instrument and say so, or it will be disabled and then ignored.
6. 06Make the cheap escape from a rule expensive. If admitting ignorance is one word, everybody writes the word.
7. 07A check that fails at random is not a check. Chase the flake before trusting the signal.
8. 08Assert a lower bound on what the walker found. Four green checks over an empty list look exactly like four green checks.

For clarity about what this is: the failure line is in the script that is in the repository, and it has since run over merges that were green. It has not yet been proven by a red run in production use, only by the summary it was written against. The typing figures are from one measurement on one day and the file count moves every night.

From the platform work on Provibr. Another guard that had to prove its own zero is in an accessibility exception that measures itself, and a field that quietly held an intention instead of a fact is in a date in the future is not a schedule.