Every self call was in tail position

Our automation flows are graphs. A trigger fires, the walk visits nodes, each node does something, and at some point the walk runs out of wires and the run is finished. Until this week that walk was a recursive function that called itself once per outgoing wire, and that was fine for as long as a run finished inside the request that started it.

An approval node breaks that assumption completely. The run has to stop, be written down, and continue tomorrow when somebody clicks a button. Half a JavaScript call stack is not something you can serialise into a database column.

So the walk became a stack machine. The interesting part is not that it became one; it is that the rewrite was provably the same walk rather than a new one that hopefully behaves the same.

Why the rewrite is exact and not approximate

Every recursive call in the old walk sat in tail position. The shape was literally await follow(node, port); return;. Nothing ever happened after the call came back: no accumulation, no cleanup, no second half of a function waiting for its child to finish.

That single property is what makes the conversion mechanical. A depth first walk whose recursion sits in the tail is by construction identical to an explicit stack onto which you push the targets in reverse order. There is no design decision in there to get wrong, no question of what order to visit siblings in, no risk that a node that used to be visited after its subtree now gets visited before it.

The consequence we cared about is that the visit order is unchanged. The existing walk tests, forty three of them, were not touched and stayed green. That is the whole argument: the tests describe the old behaviour, they were written before anyone knew this rewrite was coming, and they did not need a single edit.

If the recursion had not been in tail position, this would have been a rewrite with a compatibility argument attached, and compatibility arguments are the kind of thing you believe until a customer's flow does something unexpected on a Tuesday.

What the stack buys that recursion could not

Two things, and both are impossible with a call stack.

The first is interruption. The walk state is the entire walk expressed as JSON: the stack, the visited nodes, the accumulated variables, the counter and the open loops. It goes into a jsonb column on the wait row and comes back out unchanged. A run that is paused is a row, not a suspended process, so a deploy, a restart or a crashed container does not lose it.

The second is loops. A foreach or a repeat pushes a waypost onto the stack and then follows its body. When the stack drains back down to that waypost, one pass is finished and the loop decides whether there is another. The graph therefore stays a directed acyclic graph, there is no wire that points backwards, and everything that hangs off acyclicity keeps working: reachability, the absence of cycles, the topological list view of a flow.

That last point is worth dwelling on. The obvious way to add loops to a graph editor is to let the user draw an edge back to an earlier node. It is also the way that quietly destroys every other guarantee the editor makes, and you find out when a validation you wrote a year ago starts timing out.

Idempotence in three layers that each do something different

A paused run gets woken by a worker over the network. Workers retry. So the question that decides whether this feature is safe is what happens when the same run is woken twice.

There are three mechanisms, and they are not redundancy. Each one catches a different thing.

The claim is the real boundary and the other two are structure. The reason is that the side effects of a flow, a command sent to an agent, a notification, an outbound HTTP request, happen during the walk and not after it. So the only correct place to stop the second worker is before it starts walking, not when it tries to write its results. Whoever gets zero rows back from that update does not resume, and that is the end of it.

A retry that fails is a separate problem from a retry that duplicates. There is a ceiling of five attempts with a backoff of one, two, four and eight minutes, because a walk that throws on resume, a template that was deleted, a server that no longer exists, would otherwise be retried on every tick forever with a queue table that never drains.

One column steers the queue

The worker asks exactly one question: which waits are due. That is a status and a wake time, and nothing else.

A deadline is a wake time in the future. Making a decision is setting the wake time to now. There is deliberately no second route into the executor, no arrangement where the server action resumes the run itself and the worker also might. Two routes would be a race between them, and the race would be invisible because both paths produce a correct looking run.

This also made the second kind of wait cheap. When a durable delay arrived, nothing about the queue changed. It is a row with a wake time and no human attached to it.

Two tables, because they have different lives

The machinery and the product are separate tables, and the temptation to merge them is strong because on day one they are one to one.

The machinery row answers "when should this run continue, and with what": a continuation, a wake time, a claim, an attempt counter. It is internal, it is generic, and it is deleted when it has served its purpose.

The product row answers "what was asked, of whom, and what was decided": a heading, an explanation, a deadline, a decider, a timestamp. A customer reads it. An auditor reads it back a year later. It outlives the run it belonged to.

Merging them gives you a table where half the columns are null as soon as a second kind of wait exists, which is exactly what happened when the durable delay arrived a day later.

Small decisions that are easy to get wrong

The node is called approval and not approval.wait. A dot is a namespace separator in our translation catalogues, so a node kind containing one produces a key that resolves to a nested object instead of a string. That is the kind of constraint that lives in a different system from the one you are editing.

Expiry has three outcomes and not four: approve, reject, or skip. Skip is the one that needed thinking about, because the obvious fourth choice, "leave it hanging", is not a choice, it is the absence of a policy. A skipped approval takes neither branch and says so in the ledger, which is different from being approved and different from being rejected.

Deciding is its own permission rather than a property of the module. Without that, the person who writes a flow signs off their own approval steps, and the whole node is decorative.

A loop body is a closed area. You cannot end a run from inside one, and a wire from inside the body to something outside it is a validation error rather than a surprise at runtime. Nesting stops at three levels, iterations at one hundred per loop and five hundred visits per run, and going over any of those is an error rather than a truncation. Truncating means a customer's flow silently did half its work.

A repeat will not start on a path that does not exist yet, because a missing value reads as false, including for a not-equal test. That last clause is the one that catches people: intuitively "this field is not equal to done" should be true when the field is absent, and treating it that way starts loops on data that has not arrived.

Three gaps that the round found rather than made

The first is a refusal code for an action node without a server that had existed since the round that introduced it and was in none of the nine translation catalogues. Nobody had hit it. There is now a guard that reads the reason codes out of the source files rather than from a hand written list, and demands a sentence for each of them in each of the nine catalogues. It went red the first time it ran, with 83 findings.

The second is that the test files for this whole subsystem were not covered by any type checking configuration. They compiled in nobody's build. Adding them found six type gaps, one of which was a run summary status typed as a plain string, which means every comparison against it was unchecked.

The third is that the log node's behaviour was not described anywhere on the run page, so a customer looking at a finished run saw a step whose purpose was unexplained.

None of these were caused by this work. All three were found because writing a new node kind means reading the surrounding code properly for the first time in months.

What we would write down

1. 01Check whether your recursion is in tail position before you argue about a rewrite. If it is, the conversion to an explicit stack is mechanical and your existing tests are the proof.
2. 02Tests you did not have to edit are the strongest evidence you can get. Forty three unchanged green tests say more than forty three new ones.
3. 03Serialise the continuation, not the process. A paused run should be a row that survives a deploy.
4. 04Loops in a graph editor do not need a backward edge. A waypost on the stack keeps the graph acyclic and everything that depends on acyclicity keeps working.
5. 05Idempotence belongs before the side effects, not after them. Claim the work, then do it.
6. 06One column should steer the queue. Two routes into the same executor is a race you will not see in testing.
7. 07Machinery and product are different tables as soon as there is a second kind of machinery, which will be sooner than you think.
8. 08An expiry policy with no default is not a policy. Name the outcome, including the one that means neither branch.
9. 09Approving is a permission, not a side effect of being able to edit.
10. 10A limit should be an error, not a truncation. Half a flow that reports success is worse than a flow that stops.

For clarity about what this is: the merge verification ran 16 of 16 steps without a failure line, the end to end run against the development database was 83 green, the browser test with a real gateway worker was 27 green where the only human action is a click and the worker logs that it woke one and resumed one, the canvas test was 37 green and the accessibility sweep found nothing across five pages. Not proven: a production build, the attempt exhaustion and stale claim branches, the repeat and break nodes outside the pure test suite, and loops containing an action node.

From the platform work on Provibr. What the same graph had to guarantee before any of this existed is in a flow is a graph that has to be finite, and the editor it is drawn in is in a wire you can drag and exactly one trigger.