Skip to content

Blog ·

A capability word for an old command

When a round adds a protobuf field, an old agent fails loudly. When it only changes what a flag means, the same agent answers plausibly and is wrong.

By Provibr

Our agent runs inside the customer's own network and dials outward to us. It updates itself, but not instantly and not always, so at any given moment the fleet is a spread of versions and the platform has to know what each one can do.

It knows because the agent says so. On every session the agent sends a list of short words, one per capability, and the platform refuses a command whose word is missing rather than sending it and hoping.

This round added the third such word for Proxmox containers, and it is the first one where the interesting question was not whether to add it but what would have happened without it.

Why the words exist at all

The mechanism protects against a specific property of protobuf: an unknown field in a oneof is discarded.

When a round adds a new command as a new field, an agent that predates the round parses the message, does not recognise the field number, drops it, and is left holding a ProxmoxCommand with no command inside. It then says exactly that. The message is honest and completely useless: the operator sees a complaint about an empty envelope and the cause is four layers away, in a version mismatch nobody mentioned.

So the platform refuses first. If the word is missing, the command never leaves, and the refusal names the capability and the agent's version. That is a worse outcome than the command working and a much better one than the command failing in a way that takes an afternoon to trace.

It is a loud failure either way. That turns out to matter.

The third word is not a new field

The change this round was that a container can now show its hardware. The command that does it, GetHardware, is not new. Every agent going back to the early virtual machine lifecycle work knows the message and knows how to answer it.

What changed is a flag next to it that says which kind of resource the identifier refers to. An agent from before this round receives that flag, does not know it, and drops it, exactly as protobuf says it should. Then it does what it has always done with a hardware request: it asks Proxmox for the machine's configuration on the QEMU path.

Nothing is empty. Nothing complains. The agent believes it has been asked a question it understands, and it answers.

There are two ways that lands, and neither of them is loud. If Proxmox has nothing at that identifier under the virtual machine path, the customer is told that a machine does not exist, about a container that is running in front of them. And if a configuration does come back and gets handed to the parser built for virtual machines, the answer is not empty, it is wrong.

The wrong answer is specific and it looks fine

The two config formats are close enough to parse and different enough to matter. A virtual machine writes its first network card as a model and a MAC address joined by an equals sign. A container writes an interface name first, then the MAC in a separate keyed field.

The parser for virtual machines takes the first key and value pair as the model and the address. Given a container's line, it therefore reports a network card whose model is name and whose MAC address is ETH0.

That is a plausible looking row in a table. It has a model, it has an address, it is rendered without a warning. It is the reason the container path needed its own parser, and it is exactly the class of defect that nobody reports, because there is nothing in it to report. Somebody looks at a screen, thinks the model column is odd, and moves on.

So a command that already exists is a more dangerous kind of change than a command that is new. The new one fails at the envelope, in public. This one fails at the meaning, in private.

Why widen nothing and add one line instead

The obvious alternative is to widen the word that already exists. There is a word saying the agent understands the container path at all, and it would have been one character of work to make it mean the newer thing as well.

That would refuse every container command from every agent that has not yet updated, including the ones that work perfectly today. A fleet that is currently serving containers would go dark for a capability it genuinely has, because we changed what our word meant underneath it.

A word is a promise about what has been built, and the promise has to keep its meaning for as long as anybody is still holding it. Adding a third word costs one constant on each side and refuses exactly the commands that need refusing, and nothing else. It also puts the reasoning somewhere a person will find it, which is next to the constant rather than in a commit message.

The two sides are two constants, one in the Rust source and one in the shared TypeScript, each naming the other. That pairing is not elegant, and every attempt to make it elegant ends with a build step between two languages that need to be readable independently.

The stub says two of the three on purpose

There is a stub agent that lets us drive the gateway without a hypervisor, and it announces the container words too.

It announces two of them. It says it knows the container path and it says it reads container hardware, and it deliberately does not claim the third, because reinstalling a container is something it cannot do.

That is worth stating because a stub that claims everything is a stub that makes every refusal path untestable. And the opposite matters just as much: without the hardware word the gateway would have refused, correctly, and the check would have quietly measured a refusal instead of an answer. A test harness that lies about its own capabilities produces green results about the wrong thing.

What we would write down

  1. Protobuf drops unknown fields silently, so a new command reaches an old agent as an empty envelope.
  2. Refuse before sending. A refusal that names the capability and the version beats a complaint from four layers down.
  3. A change with no new field is the dangerous one. Nothing is dropped, so nothing complains.
  4. The failure mode you want is loud. A plausible answer that is wrong will never be reported to you.
  5. Two config formats that both parse are worse than two that do not.
  6. Never widen an existing capability word. It is a promise held by agents you cannot reach.
  7. One extra word costs one constant on each side, and refuses only what needs refusing.
  8. Put the reasoning next to the constant, where the next person will actually be standing.
  9. A stub must decline what it cannot do, or every refusal path becomes untestable.
  10. A harness that overclaims produces green results about the wrong thing.

For clarity about what this is: a third capability word for Proxmox containers, added alongside a change that introduced no new protocol field, with the reasoning recorded against both constants and the stub agent announcing two of the three; the round moved the container test module from thirty five tests to thirty nine, formatting and the release target clean, and finished with the shared packages at 5347 green, the gateway at 1467, the stub tools at 489 and the web suite at 4593, with no failures, plus fifty browser checks green in two languages. Not proven: nothing here was run against a real Proxmox, so the behaviour of an older agent on a container identifier is established from the two config formats and the parser rather than observed, and the chain in operation was exercised against our own stub.

From the platform work on Provibr. What happens when two rounds reach for the same field number is in two rounds, one field number, the round that first taught the agent about containers is in reinstalling a container is a replace, and how an agent decides whether to trust a new build at all is in how a signed agent update refuses itself.

Start with one host and one agent

Create an account, install the agent with a single command and connect your first Proxmox host. The trial runs for 14 days and asks for no payment details.