Reinstalling a container is a replace

Reinstalling a virtual machine on Proxmox is almost boring. Stop it, delete it, clone the template over the same ID, let cloud-init do the rest. The template carries a disk and the disk carries an operating system, so "reinstall" is really "make a new copy of a known good thing".

A Linux container has neither. It is unpacked from a tarball, and everything it is after that, the cores, the memory, the swap, the root filesystem size, the network interface, the resolver, the hostname, lives in a config file on the hypervisor and nowhere else. There is no image to clone over. So a container reinstall is a destroy followed by a create, and once you say that out loud a set of obligations follows that a VM reinstall never has.

This is how we built it, and the part that took the longest was not the destroying.

Two phases, because the platform does not know what is mounted

The flow is split. The first phase only reads, and changes nothing: the agent fetches the container's configuration and hands it back. The second phase stops, destroys, creates and starts, using what the first phase read as a precondition.

The obvious alternative is one phase with a preview built from what the platform already knows. That is not available here. Hardware inspection for containers is not implemented, because the parser it would use reads QEMU configuration keys, so there is no stored snapshot of a container's mount points anywhere in our database. A preview that labels every mount point "unknown" is not a preview, it is a disclaimer.

Five words for what happens to the data

Every mount point in the configuration gets exactly one label, and the label is about the data, not about the mount.

The last row is in the list and is never produced. Keeping a volume across a destroy means detaching it, deleting the container without collecting unreferenced disks, and reattaching it afterwards: three steps whose middle one, on failure, leaves a volume nobody owns any more. That is a separate piece of work with its own cleanup loop. The value exists in the vocabulary so that a later version does not have to quietly file it under a different name, a function states that it is not offered yet, and a test asserts that the classifier never returns it.

The important distinction is the one in the first sentence. A mount point can have its data survive and still not come back, because the create call we send only ever sets the root filesystem. Every other mount disappears from the new container regardless of what happened to its bytes. Getting that backwards produces the most dangerous sentence in this whole feature: "preserved" next to a mount that will not exist afterwards.

Ownership is in the volume identifier

Proxmox writes a volume in one of two shapes, and both of them say which guest owns it. local-lvm:vm-5001-disk-0 and local:5001/vm-5001-disk-0.raw both belong to container 5001. A bind mount is neither: it is an absolute path, and the leading slash is what Proxmox itself tests on.

That parse decides excluded against erased. A mount point in this container's config that points at another guest's volume is skipped by the destroy, because Proxmox checks ownership, so that data is not wiped. Without the comparison the preview would say "erased" about data that stays. Somebody would then restore a backup they did not need, which is a smaller disaster than the reverse but still one we caused.

The preview is a precondition, not a picture

What the customer saw travels back with the apply command as a list of acknowledgements, and the agent reads the configuration again immediately before destroying. If it differs, a mount point added, a different volume under the same key, it refuses.

Between the preview and the click there is a human being, and in that time an administrator can attach a disk in the Proxmox interface. "Bind mounts and host managed mount points are not automatically part of a reinstall" is only true if something enforces it.

Each acknowledgement is a key and a volume, and deliberately not the label. The label is the platform's judgement and has nothing to decide at the far end; if the agent were handed it as part of what it compares against, there would be a branch that chose on it, and then the rules would live in two places. The label is still written down, next to the acknowledgements in the command record and in the intent row, because that is the audit side, and there it should say exactly what the customer was shown.

Grow only, compared in bytes, shown in gibibytes

A reinstall may not shrink the root filesystem. The comparison runs on bytes and the screen rounds up to whole gibibytes.

Rounding up is the conservative direction, and the counter-example says why. Round 8.5 GiB down to "8 GiB" on the screen, accept a request for 8 GiB against it, and half a gibibyte is gone that nobody asked to lose. Measured against a rebuilt Proxmox, a container reporting 16 GiB where the plan asked for 8 was refused.

There is a trap one layer below this. Proxmox writes sizes as 8G, 512M or as a bare number, and a bare number is kibibytes. Read as bytes, 8388608 is 8 MiB; read correctly it is 8 GiB. A parser that guesses wrong here does not throw, it just approves every shrink it is ever shown.

Nine reasons to refuse

Before anything irreversible happens there are nine named blockers: the container is gone, protection=1 is set, Proxmox holds a lock on the guest, one of our own backup or restore jobs is running, a protected restore point exists, the root filesystem would shrink, its current size could not be read, a mount point could not be classified, or a reinstall is already in flight.

Each one is a typed value with its own translated sentence in nine languages, so a reason that is not on the list cannot get one. They are checked twice on our side, once when the preview is drawn and again when the confirm button is pressed, and a third time by the agent against the configuration it reads itself. Three closures, because this is the irreversible step of the chapter.

The root password is asked in the second phase, not the first, and travels as transient data rather than being stored. A secret you type just before the irreversible step does not spend half an hour sitting in a browser tab.

Needs reconcile is the third outcome, and it keeps the lock

If something fails between the destroy and the create, the run does not become "failed". It becomes "needs reconcile", carrying the container ID, the step it died on, and whether the container still existed when the agent gave up. That last field can be null, and null is not false: "I could not determine it" must not read as "the ID is free".

Whether a failure is destructive is derived from the position of the step in the ordered stage list, from the destroy onwards. That is why the stages are a list and not a set of loose strings: the question "had anything irreversible happened yet" gets answered in exactly one place.

A run in that state keeps the lock on the server closed. Starting a second replacement on an ID where nobody knows whether a container is sitting is precisely the thing this design forbids, and a human has to close it out.

The loser of a compare and swap must re-read

There is no worker updating these rows. The state advances when somebody reads it: the read looks at the command the intent is attached to and moves the row forward, with a compare and swap on a generation counter, so two simultaneous readers together perform each transition exactly once.

That much was designed. What was not designed, and what the browser test caught, is what the loser of that swap should return.

The panel polls once a second after a preview is requested, and that poll performs the same transition. If the customer loads the page in that same second, the poll wins the swap, the page hands back its own, now stale row, and the screen keeps saying "waiting for the agent" while the preview is already in the database. One navigation, one request, and a screen showing a state that no longer existed. Only a second reload fixed it.

The fix is one sentence long and worth internalising: losing a compare and swap means, by construction, that somebody else just made exactly the transition you wanted. The correct answer is therefore their row, and it costs one select. Returning your own is the intuitive thing to do and it is always wrong.

What we would write down

1. 01If the thing you are replacing has no image, call it a replace. The word changes what you owe the customer before you start.
2. 02Label the data and the mount separately. They are two questions and the combination of a true answer to one with a false answer to the other is the worst output the system can produce.
3. 03Make the preview travel back as a condition, re-read at the far end, and refuse on any difference. A human clicked in between.
4. 04Round conservatively when a rounding error can only destroy.
5. 05Give a half finished destructive operation its own state, keep the lock closed, and never let it fall into either success or failure.
6. 06Re-read after a lost compare and swap, always.

Container support on Proxmox is being rolled out in stages, and reinstall sits late in that order for the reasons above. If you are earlier in that sequence, the trade offs between the two guest types are in VM or container on Proxmox, and the reason a successful API call is not a finished job is in why a Proxmox 200 is not a result.

From the Proxmox work on Provibr.