Nowhere is a place: moving hardware between sites

Moving a server from one rack to another inside the same building is one physical act with one uncertainty: is it there. We modelled that a while ago, and the interesting part turned out to be the instant when the destination is neither reserved nor occupied.

Moving a server to a different city is not the same problem wearing a bigger coat. It is a sequence of acts in two places, with an interval in between during which the machine is in no datacenter, plus a network side, an address side and an agent side that all have to become true again at the other end. The temptation is to reuse the existing move and drop the constraint that forces origin and destination to be the same site. We did not, and this is why.

Why it is a second object and not a flag

The existing move already has an origin, a destination, a frozen preflight, a hold on the destination, a compare and swap counter, and an append only trail. What seems to be missing is one database check.

Adding a flag would fuse two state machines into one that fits neither. A same site move has nine states; a relocation has fifteen. A same site move has two ways out of "we do not know"; a relocation has three, and the third one is that the machine is lost in transit, which cannot happen between two racks in the same room. And dropping the check would silently allow every existing move, including ones planned last week, to cross a site boundary.

What is shared is everything that genuinely is the same: the same occupancy primitive with its partial unique index, the same append only location history, and the same operation seam of start, advance and finish. The event kinds are separate, so that a reader a year from now can tell whether a machine shuffled inside a hall or spent an afternoon in a van. One merged "moved" entry would erase that difference permanently.

Fifteen states, and one of them is nowhere

Three administrative states before anything physical happens (draft, planned, approved), nine phases of actual work, and three terminal ones. The phases are: shutting down, unracked, in transit, destination claimed, racked, recabling, reconciling, verifying, and needs reconcile.

The state that carries the whole design is in_transit. It is the only running state in which the machine's site is genuinely empty, and the code writes that emptiness rather than describing it: the site column really does go null. Null on a site normally means "not established", and here it means something stronger and more literal. There is no datacenter it is in. It is on a road.

Its neighbours are deliberately not that. unracked means out of the rack but still in the building, on a cart, so the site stays the origin. destination_claimed means it arrived at the far end but is not in a rack yet, and from that point on it administratively belongs to the destination. Two boundaries, an hour apart, and collapsing them would make the answer to "where is machine 41 right now" wrong for one of them.

From shutting down there is no way back

Transitions are a record over all fifteen states rather than a switch statement, for the same reason as everywhere else in this codebase: a switch with a default turns a newly added state into a permitted one, and a state somebody adds because something unusual is happening would be the first state allowed to do anything.

The row that matters leaves approved. Up to and including that state a relocation can be cancelled and nothing has happened to anybody. From shutting_down onward, cancel is gone. Somebody has started winding down a customer's running service; a database update does not undo that, and pretending otherwise is a claim about the physical world that nobody verified.

What remains from every physical phase is the next phase, "we do not know", and "it failed". Every physical phase can reach "we do not know", which is a difference from the same site move, where only one state can. The reason is scope: a same site move has one moment where nobody is sure, and a relocation has six. Is it still in the old rack. Is it on the cart. Is it in the van. Is it on the new floor. Is it in the rack. Is it plugged in.

Four things it notices and never fixes

A relocation writes exactly two things: the machine's site and location, and the rack claim. Not an address, not an agent membership, not a VLAN, not a fabric, not a topology edge.

That restraint is the actual requirement, and it is the one where the wrong answer looks obvious. The machine went to the other site, so surely its address prefix should follow. It should not. An address out of a prefix routed in the origin site does not work in the destination, and which address should be there instead is not something the platform knows. Which agent serves the new hall is a person's decision. Confirmed evidence is the only thing allowed to create topology truth.

So each of those becomes an open check: measured, recorded, and displayed, never executed.

The last two are honest about being permanently open rather than being quietly marked clear. Calling something clear is a statement, and nobody made it.

This restraint is guarded by shape, not by discipline. A test walks the syntax tree of the six files in this flow and fails on any insert or update into an address, agent, network, fabric, service or server table. It was proven red first with a planted write. A second, older test asks a different question, namely whether a site value came from a calculation, and it would happily approve a relocation that passes a destination site along correctly. Two questions, two tests, and only together do they cover the rule.

Three ways out of not knowing

When the physical outcome is uncertain, somebody has to go and look. All three answers are recorded as observations rather than rollbacks.

Seen at the destination. The relocation finishes, and the trail says "observed at destination" instead of "completed", so it stays readable that there was doubt.

Seen at the origin. The reservation at the far end is released and the relocation ends as failed rather than cancelled, because somebody did physical work and it did not achieve the move. The old placement is deliberately not reopened. If the machine was already out of the rack, those units were freed and something else may be hanging there now; reclaiming them could put two machines on one unit, which is the exact failure the partial unique index exists to prevent. The machine belongs to the origin site and stands in no rack, which is the truth: it is on a cart. Putting it back is a new placement and therefore its own act.

Not found anywhere. The destination reservation is released, because holding rack space for hardware that is not coming blocks somebody else for nothing. The machine stays in transit with no site, and the operation ends failed with its own blocker. It is emphatically not retired or quarantined. Those are decisions about the machine, and "lost" is an observation. Hardware that turns up next week should not have disappeared from every list in the meantime.

The preflight order, and one thing about sites

Planning refuses in a fixed order: the asset exists, there is no move and no second relocation running (two tables with two partial indexes that know nothing about each other), it is actually placed somewhere, the two sites differ, the destination site is active, no service is in the way, and the space fits.

Only the destination has to be active. An origin that is winding down is not an obstacle, it is the reason people relocate hardware in the first place. Getting that backwards would block exactly the migration the feature exists for.

The service check refuses on reserved and provisioning as well as on active, which surprised us and then did not. A machine with a reservation on it is spoken for; the thing to cancel there is the order, not the machine's journey.

What we would write down

1. 01A different shape deserves a different object. If your new case has six more states, one more exit and a period of being nowhere, a flag on the old one produces a machine that describes neither case.
2. 02Write "nowhere" down literally. A null that means "in a van" is more useful than a state that pretends the machine is still somewhere.
3. 03After reported physical work, never roll back. Add a state that means "unknown" and give it documented exits, all of them observations.
4. 04Do not reopen a freed slot. Space you released may be occupied. Going back is a new placement, with the same collision check as any other.
5. 05Notice without fixing. An open check that a person resolves beats an automatic guess about addresses, VLANs and cabling.
6. 06Guard restraint with a test on shape. "This flow never writes to those tables" is checkable; "we will remember not to" is not.
7. 07Check the direction of your site rule. Requiring the origin to be healthy would forbid the one migration everybody actually needs.

For clarity about what this is: the datacenter logistics work sits behind a rollout flag that is off by default with a per organisation setting on top of it, so none of this is purchasable today. Four things are unproven. The migration was applied by hand on the development database and rolled back, never through the migrator. The reconcile sheet and the cancel button were not driven in a browser. A raw request against the actions with the gate closed was not measured. And a destination hold has never actually been observed expiring.

From the platform work on Provibr. The same site version of this problem, and why a hold becomes a placement instead of being released and rewritten, is in the hold becomes the placement. Why a rack's free units are not the same thing as capacity is in rack space is not capacity.