Skip to content

Blog ·

The hold becomes the placement

Moving a machine between two racks has one dangerous instant: the moment the destination is neither reserved nor occupied. Here is how we removed it.

By Provibr

Moving a server from one rack unit to another is, physically, a short job with a screwdriver and a second pair of hands. The administration around it is where the interesting failure lives, and it lives in one instant: the moment when the destination is no longer reserved and not yet occupied.

If somebody else's write lands in that instant, two machines are recorded in the same slot. In a database that reads as a constraint violation you can argue about. On the floor it reads as an engineer standing in front of a rack with a machine in their hands and nowhere to put it.

The window, and how to not have one

The obvious sequence is three steps. Release the old claim, release the move hold on the destination, write the new placement. Inside one transaction that window is invisible to other sessions, which is exactly why it is easy to leave in: it will pass every test you write. It is still a window, because the statement after it operates on a slot that the previous statement freed, and one wrong ordering during a later refactor is enough.

We do something simpler. The move hold is promoted rather than replaced. The rows that record which units are occupied stay exactly where they are; only the header changes kind, from a move hold into a placement, and its expiry is set to null in the same update. The old placement closes in the same transaction. There is no point at which the destination is free, so there is no second chance to collide, and the whole thing is one update rather than an insert that has to succeed.

The claim identifier is unchanged through the promotion, which is the way to check that this is really what happened rather than something that merely looks like it from the outside.

Setting the expiry to null is not tidiness. A database check refuses a placement that carries one, because a placement that quietly stops counting on a Tuesday while the machine is still bolted in is a rack the software believes has a free unit in it. An expiring reservation is a feature; an expiring placement is a lie with a timer on it.

The arbiter is still one index

None of the above is the thing that prevents a double booking. That remains a partial unique index over rack, face and slot, restricted to rows that have not been released, and a move hold sits under it exactly like a placement does.

Two engineers who pick the same unit at the same second both see it free in the planning function that runs before the write, and precisely one of them gets through the database. The other one's write fails on the index, and the single path that writes occupancy catches that violation and turns it into a readable refusal naming the units that clash. Which sentence it turns into is chosen by the name of the constraint that fired, because "this unit is taken" and "this machine is already standing somewhere else" send a person to two different racks. A direct insert that skips that path receives the raw violation instead. Measured both ways, because reasoning about isolation levels is not evidence.

The planning function is a message, not a defence. The index is the defence. Keeping those two roles apart is what lets the message be helpful without anybody being tempted to trust it.

Nine states, and two exits that stop existing

A move has nine states: draft, planned, approved, in progress, awaiting verification, needs reconcile, completed, cancelled and failed. Which transitions are legal is a record over all nine rather than a switch statement. A switch with a default lets a newly added state fall through to permitted, and the dangerous direction here is exactly that one: a state somebody adds because something unusual is going on would be the first state allowed to do anything.

The interesting row is the one leaving in_progress, because two exits that every earlier state has are gone from it: there is no path back to approved and no path to cancelled. Somebody has the machine in their hands at that point, and the only honest exits are "reported done", "we do not know" and "it failed". Cancelling is a decision made beforehand; after physical work has been reported, a rollback in the database is a claim about the physical world that nobody verified.

The three terminal states, completed, cancelled and failed, carry empty lists rather than no entry at all. That is the difference between "nothing goes on from here" and "nobody thought about this", and only the first is a decision.

A second table says, per state, whether the destination should be held. The entry worth defending is needs_reconcile, which keeps it. Nobody knows where the machine is at that point, so releasing the destination could put a second machine on a unit that may already be occupied.

Moving is not in transit

Assets carry a process state, and this work added moving next to the existing in_transit. They sound like synonyms and they are not. moving means the machine is being relocated inside this datacenter and its destination space is reserved. in_transit means it is on the road between two sites and is therefore nowhere. Both block delivery to a customer, but they answer different questions when somebody asks where a machine is, and merging them would make the answer to the second question wrong. The separation is enforced and not merely described: a database check requires the origin site and the destination site of a move to be the same one, so a move is by construction something that happens inside one building, and the road between two sites is a different process with a different word for it.

Which state the machine had before the move is stored in the frozen preflight, not assumed. Hardware that just arrived sits in staging, and restoring everything to normal afterwards would quietly sign every relocated machine out of intake.

Two ways out of not knowing, and neither is a rollback

When the physical outcome is uncertain, the move goes to needs_reconcile and somebody has to go and look. There are two answers, and both are recorded as observations.

ObservationWhat happens to the moveWhat happens to the space
Seen at the destinationCompletedThe hold becomes the placement, as above
Seen at the originFailed, with a reason saying it was never executedThe hold is released; the original placement is untouched

The second one is deliberately failed rather than cancelled. Somebody did physical work and the machine did not move; that is an outcome, not a choice. And it is not a rollback: the database is not returned to a state pretending nothing happened. The location history is append only, corrections are new compensating entries, and the difference shows up a month later when somebody asks why a machine sat still for a day. A rollback leaves nothing behind. A compensating entry shows that there was an uncertainty and how it was settled.

Those history rows carry site and rack identifiers without foreign keys, on purpose. A rack that is retired next year does not change what stood in it last March.

Two small decisions that will matter later

The move points at its background operation with a reference that is SET NULL when the operation is deleted, rather than CASCADE. Operations are pruned after ninety days. Hardware administration is not, and a cascade would mean the retention policy for progress bars silently deletes the record of where a machine went.

And a partial unique index allows one open move per asset. Without it, two people can plan two different destinations for the same machine and both look correct on their own screen.

What we would write down

  1. Promote, do not re-create. If a reservation and an allocation are the same shape, change the kind rather than releasing and writing, and check that the identifier survived.
  2. Keep the arbiter in the database and the explanation in the code. The index decides who wins; the planning function says which units clash.
  3. Write the transition table over every state. A default branch turns a new state into a permitted one.
  4. After reported physical work, never roll back. Add a state that means "unknown" and give it two documented exits, both of them observations.
  5. Make the history append only and let corrections be new rows, because the fact that something was once uncertain is itself information.
  6. Check what your retention policy is attached to. A cascade from a short lived table into a long lived one is a data loss bug with a schedule.

For clarity about what this is: the datacenter logistics work sits behind a rollout flag that is off by default with a per organisation setting on top of it, so none of it is purchasable today. Four things are unproven. The live progress event for a move was exercised without its message bus. Cancelling a move from the operations screen was not driven in a browser. A hold that genuinely expired has never been observed, and there is deliberately no sweep that clears one, because a hold that tidies itself up while somebody is standing at the rack is the exact contradiction the index exists to prevent. And the same site check described above has never actually fired, because no second site was seeded to fire it with.

From the datacenter work on Provibr. Why a partial unique index is the only real limit on occupancy is in the only real limit on rack occupancy, and why the five capacity questions above a rack cannot be added together is in rack space is not capacity.

Start with one host and one agent

Create an account, install the agent with a single command and connect your first Proxmox host. The trial runs for 14 days and asks for no payment details.