Projecting, never replacing

A hosting panel accumulates long running work. A migration between two control panels, a backup, a restore into a new machine, a container being replaced. Each grew its own table, its own status vocabulary and its own screen, which is fine until somebody asks the obvious question: what is currently running on my account, and is any of it stuck.

The tempting answer is a jobs system. One table, one status enum, everything else writes into it. The tempting answer is also how you get two sources of truth for the same fact, and a year later a support conversation where the job says failed and the backup table says the archive is on the storage.

So the operations view is a projection. It reads the existing rows and writes a summary on top of them. Nothing ever goes the other way: not one function in the mirroring module touches the backup runs, the restore runs, the migration runs or the reinstall intents. Those tables remain the truth, and the summary is allowed to be stale, wrong or absent without anything downstream breaking.

One mirror per table, and a test that reads the schema

Four tables sit on this seam, and each has exactly one function that projects it.

The last row is why the column names are written down as strings rather than assumed. That table calls the column state, and a mirror that reads status gets undefined and falls through to the failure branch without complaining about anything.

That table is not documentation. A test derives from the database schema which of those tables have a status column, demands an entry here for each one, demands that the named function actually exists, and, the part that does the real work, demands that every file which writes one of those status columns also names its mirror. A fifth table joining the seam without a projection is a red test rather than a drawer that silently stops moving.

The reason it has to be that strict is spread: a backup run changes status in six places in a single engine file, three of them on the path that handles the answers from the reconcile that runs overnight. Letting each writer call the projection itself means the sixth one forgets, and forgetting is invisible. The row is correct. The screen just never updates.

The sink that existed twice

The mirroring lives in the package that owns the database writes, because both a server action in the web app and a gateway worker call the same functions. That package cannot publish a live event: it depends on the database and on shared code and on nothing else, and the message bus is a different object on each side.

So it exposes a sink. Each application registers one, and the package does not know who filled it in. If nobody does, the database write proceeds and no live event goes out, and the drawer updates on the next page load. That is honest degradation, and the function deliberately never throws, because a missing registration must not be able to take a backup down with it.

The sink was a module variable, and that was wrong in a way that produced no error at all. Next runs its instrumentation file as its own bundle, separate from the route workers. The variable therefore existed twice: the registration landed in one copy and the emitting function read the other. Rows were correct, statuses advanced, the drawer sat still, and there was not one line in any log.

This was not found by reasoning about it. It was found by putting a listener on the message bus and counting: one event arrived where three should have. The fix is a slot on globalThis behind a Symbol.for, which survives a bundle boundary inside one process, the same shape and the same reason as the shared database client and the shared Redis connection.

Two lessons that are worth more than the bug. A failure with no error line needs an instrument outside the code you suspect. And "this module has one instance" is an assumption about a bundler, not a property of the language.

The same round found a second silent one. Finishing an operation and blocking an operation both left the stage column alone, so a backup that completed perfectly stayed on "archiving" forever. Nothing failed; the last step simply never arrived.

Cancel is a request, and one UPDATE is the proof

Cancelling is where a projection is most tempted to lie. The drawer knows the operation, the operation has a cancel button, and the cheapest implementation sets a flag and closes the sheet.

Every kind therefore declares whether cancellation is even meaningful. A running vzdump cannot be cancelled by us: the command is with the agent, the task is with the hypervisor, and nobody knows what half an archive on the customer's storage amounts to. A restore is the same, with one extra reason: the restore row holds a lease on the target machine id for as long as it is open, and closing it early releases that id while a guest may be landing on it. Both are declared unsupported, and there is no path that overrides that.

Where cancelling is supported, the row still decides. A migration can be cancelled while it is planned and not once a helper is writing servers on somebody's host. A container reinstall can be cancelled up to the preview and not once the agent has started replacing.

When it does work, the proof that nothing happened is a single conditional update: set the queued command to cancelled where its status is still pending. If it has already been sent, that statement touches zero rows and the request honestly stays at "asked for". Checking and then writing would be two statements and therefore two chances for something to slip between them.

And then the sentence matters. The event says the command was taken out of the queue before the agent received it, not that it was cancelled. Nothing was interrupted, because nothing had started. That row goes into the general event log rather than into operation progress, because progress is swept after ninety days and "a person reached into the infrastructure" is not progress.

Four blockers, and none of them is a failure

Four situations put an operation into needs attention rather than failed.

A vzdump reported OK and the archive is not in the index. A backup attempt timed out and we do not know whether the archive exists. A restore cannot be read back or the resulting guest could not be recorded. A container reinstall stopped after the old container was already gone.

Calling any of these failed would be a lie with a consequence attached. Failed invites the customer to run it again, and running a backup again is another few dozen gigabytes on somebody's storage for an archive that may already be there. Running a reinstall again is a second container. The reconcile loop keeps looking until the index gives an answer, and the operation stays visibly stuck in the meantime, which is the correct amount of alarming.

The blocker also carries who owns it. Three of these are ours; a command that genuinely failed on the customer's side says so.

A dry run ends at planning, and that is not a gap

A migration in dry run mode has five steps; a real one has eight, the extra three being writing and counting. The stage list has all six stages, so a successful dry run shows its last two as not done.

That looks like a bug in a progress bar and it is the truth: a dry run does not write and does not count. The alternatives were a second operation kind, which duplicates nine catalogue entries in nine languages for the same work, or a stage list that varies per row, which makes the drawer and the detail page disagree about how many steps there are. Showing the truth was cheaper than both.

What we would write down

1. 01A projection projects. If your summary layer can write back into the source, it is a second system of record and you now have two.
2. 02Put the seam in a table a test can read, derived from the schema, and make it check that every writer of a status column names its mirror. A forgotten call site produces no error.
3. 03Do not assume a module has one instance. A framework that runs part of your code as a separate bundle will give you two, and the symptom is silence.
4. 04Instrument outside the suspect. A bus listener found in minutes what reading the code had not found at all.
5. 05Say what actually happened. "Taken out of the queue before the agent received it" is longer than "cancelled" and it is the only version that is true.
6. 06Reserve failed for failure. Unknown outcomes need their own state, or people will retry expensive operations that may already have succeeded.

For clarity about what this is: the operation kinds described here sit behind rollout flags that are off by default, so migrations and customer backups are not purchasable today. Several things are unproven. The live progress event was measured for migrations only. The gateway side of the sink registration was not exercised with a gateway actually running. The agent chain was reproduced with SQL rather than an agent. And cancelling a container reinstall from the operations screen was never driven in a browser.

From the platform work on Provibr. Why a progress bar refuses to estimate a finish time is in a progress bar that refuses to guess, and why a migration that silently skipped is worse than one that failed is in the migration that never ran.