One real agent, one development gateway

Five rounds in a row ended their write up with a version of the same sentence: the protocol change is additive, the unit tests are green, and none of it has been exercised against a real agent. Monitoring adapters, the secret envelope, automatic agent updates, an automation step that waits for a command. Every one of them was correct as far as anybody could tell, and every one of them carried a caveat that the next round inherited.

That caveat is cheap to write and it compounds. Each round adds one, nobody is wrong, and after five of them there is a stack of behaviour that has only ever existed in a test harness. This round was spent removing the stack rather than adding to it: no feature, no protocol change, no release. Eighty two measurements against one real agent.

The setup, in one sentence

A real agent binary, built for the same target we ship, signed with a development key, running inside a throwaway container on a machine that is not a developer laptop, talking to a development gateway over a reverse tunnel, with seven disposable services on loopback for it to measure and a second copy of the same binary enrolled as a hosted probe.

Everything in that sentence is load bearing. The binary is the shipped one rather than a test build, because the interesting failures live in serialisation and TLS rather than in logic. The gateway is a development instance on non standard ports, deliberately, because the production gateway listens on the obvious ones and an agent that connects to the wrong side of that would be a very bad afternoon. The services are throwaway containers of the real software at real versions rather than mocks, because the entire point of the exercise is that mocks are what we already had.

The database services were seeded with a canary password, and that choice is what made the next section possible.

A grep with a planted value

The claim we most wanted to test is a negative: a monitoring secret travels sealed, and it is never written to the command row, the event log, the message bus, or either process log.

Negatives are the ones you get wrong quietly. So the passwords were not random strings whose absence proves nothing; they were planted values, chosen so that a single occurrence anywhere would be unmistakable. Then seven layers were searched: the stored command payloads, the event table, the stored check configuration, the stored results, the message bus in both text and its encoded form, the gateway log and the agent log. Together that is around half a million characters of material.

Zero hits, which is the answer we wanted and also the answer a broken search gives.

So the same search was run against a file that certainly contained the word, and found it. That positive control is the difference between "we looked and there was nothing" and "we did not look". It costs one line and it is the single most valuable line in the script.

Two more requirements guard the same failure from the other direction: the run has to observe at least a dozen real check commands and at least several dozen bus messages before it is allowed to conclude anything. Both of those were red at some point during the round, on runs that would otherwise have reported a clean sweep over nothing at all. A sweep that finds no violations in an empty dataset is not evidence, and it looks exactly like success.

Two ways a measurement is green without measuring

Both of these happened in this round, in scripts written by somebody who knows about this failure mode and writes about it regularly.

The first: a query selected one column and the code then read a different property off the result rows. In a typed language that is an error; here it is undefined, forever, silently, and every comparison against it fails in a way that looks like an assertion never firing rather than an assertion always passing. The check reported success because it had nothing to compare.

The second: a membership test against an empty list. It is a statement about nothing, it is trivially satisfiable, and it is the natural end state of any script whose earlier stage quietly produced no rows. Both are cheap to guard: assert the shape of what you fetched, and assert that the set you are about to test is not empty.

Five column names in those scripts were guessed rather than looked up. Those failed loudly, which is the cheap variant of the same mistake, but it is the same mistake: one command answers what a column is called, and the version where a guess ends up inside a list nobody executes is the version that survives review.

An agent that refuses to enrol twice

Halfway through the update tests the agent stopped coming back. From the outside it looked like a restart loop, and the first hour went into the update path.

The agent was doing exactly the right thing. Enrolment writes a state file, and the generated install script checks for that file and refuses to enrol again if it is there. Re enrolling an already enrolled host would replace an identity, and doing that by accident is how a machine loses its certificate and its vault in one step. Our supervisor loop, written for the occasion, had no such gate, and every restart attempt was hitting the agent's own correct refusal.

The lesson is not about the gate. It is that a correct refusal and a defect are the same shape from a distance, and the thing that separates them is reading what the other side actually said rather than what the symptom suggested.

Three more surprises came out of the same days, all of them design decisions meeting reality:

Scheduled update times live on a five minute grid, enforced by a database constraint. "One minute from now", which is what a person writing a test wants, does not exist. The mechanism that makes that workable is a catch up window: a slot that was missed is still executed for a quarter of an hour afterwards. That is also why a repeated measurement inside the same quarter hour has to pick an unused grid time, since a slot is claimed uniquely per agent and a second run against the same one is correctly refused.

A TCP check defaults to TLS on. Pointed at a plain port, it fails with a TLS error, which is right and which reads as a broken check until you look at what a default is.

And the audit row of an automation step whose wait expired freezes on the status the flow saw. The command became a timeout sixty milliseconds later and the row still said what it said. That is not a bug in the row, it is a bug in the comment above it, which promised you could read the eventual outcome there. That promise was removed and answered properly elsewhere.

A notification that groups, and a count that says zero

The sharpest of the small findings. After a successful update the platform writes a notification. The measurement counted notifications created since the run started, and got zero.

Notifications group. A second occurrence of the same thing increments a counter on the existing row and leaves its creation timestamp alone, which is exactly what you want in a panel: "this agent went offline" with a count beside it, rather than forty rows. It is also exactly what makes a count on the creation timestamp report nothing while the notification is sitting there in plain sight.

The measurement was wrong, the product was right, and the only reason we know which way round it was is that somebody opened the panel instead of trusting the query. Every aggregate has a shape, and a test that does not know the shape is testing its own assumption.

What we would write down

1. 01"Unproven over the wire" is a debt, and it compounds. One round can carry it. Five rounds is a stack of behaviour that has never existed outside a harness.
2. 02Prove a negative with a planted value and a positive control. Zero hits and a broken search are indistinguishable without one.
3. 03Give every sweep a floor. Assert the input is not empty before concluding anything about it.
4. 04A correct refusal looks like a defect from a distance. Read the other side's message before rewriting your own.
5. 05Do not guess a column name, a default, or a grid. All three are one command away, and the guess that reaches a list is the one that never fires.
6. 06Know the shape of what you are counting. Grouped rows do not have the creation timestamp you are filtering on.

For clarity about what this is: nothing shipped in this round. No product change, no protocol change, no agent release. What came out of it is that five caveats are gone and a handful of small defects are fixed, and the cleanup was counted rather than assumed: no leftover rows, no leftover containers, no leftover files.

Some things stayed unproven. The automation injection path was exercised through its core rather than through the panel, the adapters were not run from a hosted probe, and there is still no measurement against a real hypervisor.

From the platform work on Provibr. Why the monitoring secret is sealed per attempt is in a secret that travels once, and how a signed agent update refuses a tampered build is in how a signed agent update refuses itself.