A null that fell through the sieve

We were building the hardware tab for containers when a browser check produced a sentence it should never have produced: the configuration of this container could not be read, so what a backup holds has not been established.

The configuration had been read. It was in the response, complete, with every mount point in it. What had failed was our own parse of it, one layer further along, and the message blamed the wrong side of the wire.

Working backwards from there took a while, because there was nothing to work backwards from. No exception, no log line, no failing test. Just a calm sentence on a screen saying that a thing had not been measured.

An absent key and a null are not the same thing

The agent builds the mount point rows in Rust with serde_json::json!, and one of the fields is an Option<String>: the path inside the container, which comes from mp=/data in the config.

An Option that is None serialises there as null. Not as a missing key. That is the correct and documented behaviour and it is what any reader of that macro would expect if they thought about it for a second, which nobody had.

On the other side the schema said .optional(), and in zod a bare optional means the key may be absent. It does not mean the value may be null. Those are two different statements about JSON and we had written one while sending the other.

The field that made this fire on every single container is the one that is structurally always empty. Every container config has a rootfs line, and a rootfs by definition has no mp=, because it is mounted at the root. So the very first row of every observation from every real container carried "mountPath": null.

One bad field means no data at all

If the failure had been confined to that field it would have been a cosmetic defect. It was not, and the reason is the shape of the parse rather than the shape of the bug.

We validate the whole observation as one object. A safeParse on an object fails as a whole; there is no partial result. So one rejected field meant no mount list, no coverage classification and no preview of what a restore would replace. Everything downstream saw nothing, and everything downstream is written to be honest about nothing.

That honesty is deliberate and it is the reason the product does not tell you a backup is complete when it has not looked. A container config with no readable rootfs is reported as unknown rather than full, precisely so that an empty measurement can never be dressed up as a good one.

Here the guard worked exactly as designed and produced a sentence that was, in its own terms, true: nothing had been established. The lie was one level up, in the claim about why.

Nothing was red, and the reason is uncomfortable

This shipped with the round that introduced container backup coverage, and the seed data of three rounds passed over the same code without anything going red.

The reason is that the seeds were written by hand. To exercise the coverage table without a real Proxmox you write a row that looks like what the agent sends, and what you write when you mean this one has no path is to leave the key out. Which parses. Three rounds of test data therefore encoded the schema's assumption rather than the agent's behaviour, and every one of them was green.

There is no real hypervisor in the loop to correct that. The Proxmox we have access to is somebody's production machine, and no agent of ours goes on it. So the only description of the wire that both sides could have been checked against was the Rust source, and nothing was reading it.

The guard reads the producer instead of copying it

The repair is one field. The interesting part is the test next to it.

It opens lxc.rs, finds the function that builds the rows, extracts the key names out of the json! block, and then asserts two things. First, that the set of keys the agent sends is exactly the set of keys the schema knows about, so neither side can grow a field alone. Second, and this is the one that matters, that a row in which every optional field is null comes through the sieve.

That second test builds its row from the keys it just read, not from a list written in the test file. A new optional field added with a bare .optional() is red on the day it is added, without anybody having to remember this article.

There is a third test for the other direction: a row from an older agent, which does not carry the newer fields at all, still parses. Both shapes are real, they are both in the database, and a fix that only accepted the new one would have broken the old rows while looking correct.

The fix itself is a preprocess that maps null onto undefined rather than a .nullable(). That is a deliberate choice about the derived type: with nullable the type becomes string | null | undefined and every reader gains a third state that means nothing to it. With preprocess the type stays as it was, the four readers keep the code they already had, and the wire tolerance is expressed in exactly one place.

Two mistakes in the measuring, not in the product

Establishing that the repair worked produced two false greens of its own, and they are the more transferable half of this round.

The first: a schema written with .strict() silently rejected our own seed, and the screen went empty rather than wrong. An empty screen during a repair looks exactly like a screen that has not loaded yet.

The second is worse and it is a known trap here that we walked into anyway. The backup table lives in a panel that is always mounted in the document, because that is how the focus return of a dialog is made to work. So a check that reads the table with a query selector reads it whether or not it is open. Our assertion that two tabs said the same thing was passing on two tables that were both empty and both invisible.

Both were closed the same way: demand that the dialog is genuinely open, and put a lower bound on the number of rows before comparing them. A comparison of nothing with nothing is not a comparison.

What we would write down

1. 01An absent key and a null are different statements, and JSON serialisers pick for you.
2. 02A schema that only accepts a missing key has never read its producer.
3. 03Whole object validation turns one bad field into no data, which is the right design and a bad failure to debug.
4. 04The most dangerous error message is a calm one. Ours was honest about the outcome and wrong about the cause.
5. 05Hand written test data encodes your assumptions, including the one you are about to get wrong.
6. 06Let the guard read the producer. A key list copied into a test drifts; a key list parsed out of the source cannot.
7. 07Test the all null row explicitly, because that is the row your real producer sends first.
8. 08Accept the old shape too. Rows written by older agents are still in the database.
9. 09Tolerate on the wire, not in the type. Widen at the boundary and leave the readers alone.
10. 10An always mounted panel makes a DOM check vacuously green. Assert that it is open and that it has rows.

For clarity about what this is: the defect was in the coverage of every real container since the round that shipped container backups, found by a browser check rather than by any suite, repaired with one preprocess and guarded by a test that reads the Rust source; the round it was found in moved the container test module from thirty five tests to thirty nine, with musl and formatting clean, and finished with the shared packages at 5347 green, the gateway at 1467, the stub tools at 489 and the web suite at 4593, no failures anywhere, and fifty browser checks green in two languages with no accessibility findings. Not proven: none of this ran against a real Proxmox, so the wire shape is established from the source rather than from a live response, and the chain in operation was exercised against our own stub.

From the platform work on Provibr. What the coverage table is for is in what a backup does not contain, the round that first read a container config is in reinstalling a container is a replace, and another check that was green because it never arrived is in green because it could not get there.