Green because it could not get there

We keep a stub agent. It speaks the same protocol as the real one, it runs next to the gateway, and it exists so that the platform side of every command can be tested without touching a customer's hypervisor. It is a contract test with a process around it.

Until this week that stub answered the most ordinary command there is, turn that machine off, with COMMAND_STATUS_UNSUPPORTED.

Nobody had noticed, and the reason nobody had noticed is the point of this article. Every test that sent a power command to the stub was green. They were green because the command was refused, the platform handled the refusal correctly, and the assertion at the end held. The tests were not lying. They were describing a path that nobody meant to exercise.

What a missing branch does to the tests around it

The flow engine has an action node: run a command against a server, then follow the ok wire or the error wire depending on how it went. That node had been in the product since the round that introduced it, and it had tests.

Those tests could only ever reach the error wire. Not because of a bug in them, but because the only agent available to them refused every command they could send. A test that asserts "the error branch was taken" passes perfectly well when the error was manufactured by the test harness rather than by the thing under test.

This is the shape our own notes warn about, in a sentence that has been in the project handbook for months: a test can be green because it cannot reach the good path. The variant that bit us here is worse than the usual one, because it is not a case of a branch never being written. The branch was written, it works, and it had a passing test next to it the whole time.

The stub had to hold state, not just answer

Adding the five power commands was not a matter of returning success. The platform decides what a command is allowed to do by reading the server's current status: you cannot resume a machine that is running, you cannot start one that is already up. That decision lives in the platform, and it reads a column.

An agent that answers "done" without changing anything leaves that column where it was. So a test can send exactly one command and then it is stuck, because the second command is refused by the platform's own state machine, correctly, on a status that never moved.

The stub therefore keeps a power state per machine and sends the state observation back over the session, the same way a real agent does. That is what was actually missing. Not a reply, a consequence.

With that in place, a flow that loops over a list and issues two commands per pass could be run against the stub for the first time. Six commands, all reaching completed, and a loop that reached its second pass because the first pass had actually changed something.

Two things the stub deliberately does not do

It does not refuse impossible transitions with an invented error code. The boundary between allowed and not allowed lives in the platform, and the agent's reason codes are a fixed list compiled into the real agent's source. Making up a code that the real agent does not have would be testing a contract that does not exist. A stub that is stricter than the thing it stands in for teaches you things that are not true.

It does not simulate duration either. A shutdown on a real guest waits for the operating system to respond and falls back to a hard stop after a timeout. In the stub everything happens inside the tick. But the timeout and the fallback flag are carried through into the output, so a test can see that they arrived even though nothing waited for them. Passing a parameter through and honouring it are different claims, and only one of them is being made.

A second alarm clock makes a measurement random

The other repair in the same round is about the measuring, not the product.

Paused runs are woken by a worker in the gateway that polls a queue every ten seconds. Our end to end script does the waking itself, in process, so that it can assert exactly what happened. Those two are fine separately and catastrophic together: if a gateway with the waker enabled is running against the same development database, it eats the rows as soon as their wake time passes, and the script measures whichever of the two got there first.

That happened. One branch of the script failed once with nothing wrong with it, which is the most expensive kind of failure because the natural response is to go looking for a bug.

The fix is not a comment saying "make sure no worker is running". It is a function that measures it. Before anything else, the script creates a throwaway wait with an already elapsed wake time, watches it for eight seconds, and checks whether anybody else touched it. If somebody did, the script refuses to run and says why.

That is a small amount of code for a large amount of trust. An assumption written in a comment is an assumption; the same assumption written as a check that runs first is a precondition.

Retries need a ceiling and a rising pause

While the good paths were finally reachable, the failure paths got measured properly too.

A wait that throws on resume, because a template was deleted or a server no longer exists, would otherwise be retried on every tick forever, in a queue table that never drains. There is a ceiling of five attempts with a pause that doubles: sixty, a hundred and twenty, two hundred and forty, four hundred and eighty seconds. Those four intervals were read out of a real run rather than out of the function that computes them. On the fifth attempt the wait is cancelled and the run is marked failed.

The counter proof is the part that matters. With the exhaustion guard switched off, the same run stays in the waiting state indefinitely. A guard is only proven when the situation it prevents has actually been produced.

There is a second failure mode underneath it. A worker claims a wait by moving it out of the waiting state before it starts walking, because the side effects of a flow happen during the walk and not after it. If the process that claimed it dies mid walk, the row sits in the claiming state forever, and the query that looks for due waits does not see it: it asks for waiting rows. A sweep puts claims older than fifteen minutes back, and a single tick against a really running gateway released one, woke one and resumed one, with no failures.

Some small things that are only findable by running them

A delay of three seconds does not make an invalid node. It makes the whole graph invalid, because the minimum is five, and the runner returns nothing at all rather than a run with a broken node. That is correct and it is also confusing the first time, because you expected an error about a node and got silence about a graph.

The gateway's ports are named for the gateway. Setting the variables without that prefix leaves it listening on its defaults, quietly, which looks exactly like a networking problem.

A halt writes the nodes it never reached into the ledger as skipped. That is a deliberate difference from leaving them out: a run that ended early and a run that took a different path look identical if the unreached nodes are simply absent.

And an approval node was leaving orphan rows in the notifications table, sixteen of them on the development database. The cleanup for that is per run and keyed on the deduplication key, never a pattern match on a prefix, because a pattern match would take out rows belonging to somebody else's round.

What we would write down

1. 01A test that cannot reach the good path is green for the wrong reason. Ask what the test would do if the feature were deleted.
2. 02A stub that refuses everything is not a stub, it is a wall. Every test behind it measures the refusal.
3. 03A fake has to hold state, not just answer. If the platform decides on a column, something has to move that column.
4. 04Do not make a stub stricter than the real thing. An invented error code tests a contract nobody implements.
5. 05Pass a parameter through even when you cannot honour it, so a test can prove it arrived.
6. 06Two wakers on one database make the measurement random. One of them wins and you cannot tell which.
7. 07Turn the precondition in the comment into a check that runs first. Then it is a fact rather than a hope.
8. 08Prove a guard by switching it off. A retry ceiling nobody has hit is a claim.
9. 09A claimed row is invisible to the query that looks for due work. Something has to put stale claims back.
10. 10Unreached nodes belong in the ledger as skipped, or an early exit is indistinguishable from a different route.

For clarity about what this is: seven previously unproven branches were taken end to end, each with a red counter proof; the merge verification ran 16 of 16 steps without a failure line, the web suite was 4583 green, packages, gateway and tools together 7287 green, the stub agent's own suite 481 green, the end to end script 89 green, the waker branch 13 green, the approval mail branch 22 green and the loop with real actions 18 green. Not proven: nothing was run against a real Proxmox, an unreadable graph on the wait row itself, a batch larger than the batch size, and real mail delivery over SMTP.

From the platform work on Provibr. How the walk learned to stop and continue tomorrow is in every self call was in tail position, and where the line sits between a pause a request can absorb and one that suspends the run is in a pause that held the order open.