Skip to content

Blog ·

A join instead of a reconcile

An automation step froze on the status it saw. The fix was not a worker that repairs the row: that row explains a decision, and repairing it destroys it.

By Provibr

An automation flow can send a command to a machine and wait for it. If the wait runs out, the step fails and the run takes its error exit. That is correct: the flow asked to wait until the work was done, and it is not done, so nobody knows whether the next step is about to act on a machine still shutting down.

The step records what it saw at that moment. Measured in an earlier round against a real agent, it recorded sent, and the command became a timeout sixty milliseconds later. The step still said sent, a day later, forever.

The obvious reading is that the row is stale and needs repairing. That reading is wrong, and working out why took most of the thinking in this round.

The row is an explanation, not a status field

The frozen value is not a copy of the command's status that drifted. It is the answer to a different question: what did the flow know when it decided to take the error exit.

Overwrite it with timeout and the row now says the flow reacted to a timeout. It did not. It reacted to a wait running out while the command was still in flight, a different event with a different cause and a different fix. A month later, somebody reading that run would draw the wrong conclusion from a row that had been helpfully corrected.

So the frozen value stays frozen. What was actually wrong was the comment above it, which promised that you could tell from this row whether the command eventually succeeded. You could not, and a promise in a comment has a shelf life measured in the time until somebody relies on it.

The real question, "did it come good in the end", is answered by reading the command's current state at the moment somebody looks. The run page shows both, side by side: what the flow saw then, and what is true now.

Where it comes fromWhat it answers
ThenThe audit row, frozen at the moment of decisionWhy the run took the exit it took
NowThe command table, read when the page is openedWhether the work eventually finished
Row goneNeither; shown as its own stateThe command is no longer on the books

That last line matters. When the command row has been cleaned up, the display does not fall back to repeating the frozen value. "It is no longer on the books" and "it still says sent" are two different facts, and a page that renders the second one when the first is true is lying with a straight face.

Three reasons a worker loses to a join

The alternative was a background worker that writes the final status back into the audit row once the command settles. We already have workers shaped exactly like that, so this was a real choice rather than a straw man.

A worker overwrites an explanation. Covered above. The escape hatch is to write a second field beside the first, and that is honest, but it is a second truth that would be read in exactly one place: the place that is now doing the join.

A worker only repairs what happens after it. The row that started this whole investigation already exists. A worker deployed today would never touch it, nor any run from the two rounds before, because their commands settled long ago and no future event will revisit them. A join shows all of them the moment it ships, including the one we went looking for.

A worker brings a whole apparatus. Updating rows in the background needs a schedule, a claim so two instances do not collide, a drift interval, a retention answer and a failure path of its own, each of which can be wrong at three in the morning. Looking up a handful of identifiers by primary key when somebody opens a page needs none of that.

The general rule underneath: prefer reading the source over copying it, unless the copy answers a question the source cannot. Here the copy answers a different question and both are wanted, which is exactly the case where you keep them apart.

The identifier came out of a JSON column, so the query is scoped

One detail in that join is not an optimisation and not politeness.

The command identifier lives inside a JSON blob on the audit row. JSON columns have no foreign keys, so nothing guarantees that the identifier in there belongs to this account. Today it does, because our own code put it there. That is not a boundary, it is a habit.

So the lookup is constrained to the organisation as well as to the identifier. Without it, a hand edited value in that blob would read the status of somebody else's command. Small leak, easily missed, and the kind that is only ever found by someone who was looking for it.

One flow per moment, with the moment inside the key

A hosting plan could link one automation flow, in a single column. That column meant "run this when a machine is provisioned", which was written down nowhere except in the behaviour of the function that read it.

Customers wanted a flow on reinstall as well, and one on deletion. The obvious repair is three columns. It is wrong for a reason worth naming: each of those three columns could hold a flow whose own trigger says something else. A flow that runs on deletion, sitting in the column that fires on provisioning. The invariant "the trigger says when it runs" becomes breakable in three places instead of zero, and nothing in the schema objects.

So the link became its own table, with the plan and the moment together as a unique key. One flow per moment per plan, enforced by an index rather than by care.

Two details in that migration are worth keeping.

The moment is stored as text with a check constraint listing three values, not as the existing trigger enum. The enum has five values, and two of them are triggers that this mechanism never chooses: a manual run is started by a person, and a webhook run by an incoming request. An enum column would happily accept either, and then a link row exists that nothing will ever act on. A setting that silently does nothing is worse than a setting that is missing.

And the old column was dropped in the same migration that copied its contents across. Keeping it as a convenient derived read is precisely the second truth this table exists to eliminate. Every reader was found by the compiler the moment the column left the schema; there were seven. The copy runs before the drop, in one transaction, so there is no instant where a customer's configuration lives nowhere.

The copy also takes each flow's own trigger rather than assuming the old column's meaning. Rows whose flow carried a different moment would never have run under the old code; now the link a person made actually does what they thought it did.

Mail that is opt in, in the graph rather than in the settings

A flow can raise a notification, and this round let it also go out by mail. That turned out to be a question about defaults rather than about sending.

The platform's mail categories are a deny list: anything not explicitly switched off is on. That is right for the categories that already exist, all of which are things a customer needs to hear about. It is wrong for a notification whose text somebody wrote in an automation editor, because a flow that runs on every order would mail the whole team every time until somebody found the setting.

Rather than invert the list, the default lives in the graph: the box on the node is off, and turning it on is a deliberate act by the person who wrote the text. The category still exists so that mail can be switched off in the ordinary place, exactly like the others. Two mechanisms, each doing the thing it is good at.

A test run never sends mail. It writes the panel notification, which is internal and dismissable, and it sends nothing outward, for the same reason a test run never touches a machine: post arrives in real people's inboxes, and a test button that surprises anybody is not a test button.

One more small thing from the same work. The sending side was expected to be unreachable from where flows run, which is why this had been postponed. Measured, it was the other way around: the function we were routing around exists precisely because the other process cannot render mail, and it asks the process that flows already run in to do the rendering. The detour was longer than the direct route, and the direct route is one HTTP hop, one signature and one failure path fewer.

What we would write down

  1. An audit row explains a decision. Do not repair it. The question "what is true now" has a different answer and deserves its own place.
  2. A background repair only fixes the future. If the row that prompted the work would not be touched, that is a design answer, not a detail.
  3. Read the source unless the copy answers something the source cannot.
  4. An identifier out of a JSON column has no foreign key. Scope the query to the account; that our own code wrote the value is not a boundary.
  5. Put the discriminator in the key, not in the column name. Three columns is three places to store the wrong thing; one key is zero.
  6. Drop the old column in the same migration. A derived read left behind is the second truth you were removing.
  7. A deny list is the wrong default for text a customer wrote. Put the opt in where the text is written.

For clarity about what this is: all of it is built and running. The mail sending itself was proven with a logging driver rather than against a real mail server, and the plan link has not been exercised through a real order yet.

From the platform work on Provibr. How the wait and the error exit behave in the first place is in a wire you can drag, and exactly one trigger, and why a timeout is not a stop button is in a timeout is not a stop button.

Start with one host and one agent

Create an account, install the agent with a single command and connect your first Proxmox host. The trial runs for 14 days and asks for no payment details.