Two orders, one machine

A VPS plan oversells by design and nobody minds. A dedicated machine cannot: there is one of it, and if two orders both believe they got it, one customer is going to be told on a Monday morning that the thing they paid for on Friday belongs to somebody else.

The requirement is one sentence long. Reserve before side effects, and release capacity transactionally after a crash or a failure. Implementing it produced two measurements that changed the design, and one of them cost us a machine that was sitting there available.

The arbiter is an index

The rule that two orders can never hold the same machine lives in a partial unique index on the asset id, restricted to reservations in the held state. Not in the application, not in a lock the code takes, not in a comment.

That is deliberately the same shape as the occupancy index on rack units. The application layer produces a readable refusal before the collision, and the database decides who actually wins. Only one of those two can be authoritative, and it has to be the one that cannot be bypassed by a code path somebody adds next year.

Held is the only status that holds anything, and that is expressed as a table alongside the index rather than as a boolean column, because a boolean next to the status is a second truth that stays correct only as long as every writer remembers it. A test checks the table against the index.

There are four statuses and one of them is a pair on purpose. Held, converted, expired, cancelled. The tempting simplification is a single "released" for the last two, and it would make the question "did somebody cancel this, or did it just lapse" unanswerable, which is the exact question you get asked when a customer phones about an order that did not go through. A sweep quietly writing the same status as a human erases the difference between a decision and a timeout.

Locking the whole pool cost a reservation

The first version selected the candidate machines and locked them in one query with FOR UPDATE SKIP LOCKED. That reads well and it is wrong, and only a concurrency test showed it.

Eight simultaneous orders against three eligible machines should produce three reservations. It produced two. The first transaction grabbed all three rows, used one, and held the other two locked until it committed; everybody else skipped past all three and found nothing. The pool was not exhausted, it was hostage.

The fix is to lock one candidate at a time: take the first, and if somebody else has it, move on to the next. Rerun of the same test: three reservations on three different machines and five honest refusals. Two orders against a single machine give one reservation and one refusal.

That fix introduces its own risk, which is the length of the loop. With a hundred simultaneous orders the last transaction would walk a hundred candidates before winning, and a long transaction holds everybody else up. So the scan is capped at twenty five candidates, and above that, "try again in a moment" is the honest answer.

Note that skipping and failing are counted separately. A candidate that was locked by somebody else is skipped and that is not an attempt, because nothing went wrong. A candidate that lost an insert race is an attempt, and there are at most five of those.

A unique violation poisons the transaction

This is the trap the retry loop is built around, and it is easy to write code that looks correct and falls over on the first real collision.

Inside a PostgreSQL transaction, a unique key violation aborts the transaction. Every subsequent statement is then refused with a different error, "current transaction is aborted", until somebody rolls back. So a loop that catches the violation and tries the next candidate does not work at all: the second candidate's insert fails for a reason that has nothing to do with the second candidate.

Each attempt therefore runs inside a savepoint, which in our query builder is simply a nested transaction. Roll back to the savepoint, the outer transaction is healthy again, take the next candidate.

There is a related detail about how the violation is caught. Our query builder wraps driver errors in its own error type and hangs the real one on the cause, so checking a code on the top level error sees nothing. It has to walk the cause chain, and it has to look at the constraint name as well, because this table has two unique constraints and they mean opposite things: one says the machine is taken, the other says this order already has a reservation and here it is.

The refusal has to say which kind it is

The allocator builds the candidate list twice, and the difference between the two results is a message to a customer.

The first pass is the full list, with a reason for every machine that did not qualify: wrong site, wrong manufacturer, not ready, and so on. The second pass is what could actually be locked at that instant. If the full list produced a winner and the lockable list did not, then there was something to sell and somebody was a fraction of a second earlier. That is a different sentence from "there is nothing": one means try again, the other means buy more hardware. They deserve different buttons.

The frozen reason for the chosen machine goes into the reservation row, because a month later every asset is in a different state and it is no longer reconstructable why the allocator picked this one. The row says so itself.

Idempotency is asked twice, and the second time is another thing only a concurrency test found. Five simultaneous requests carrying the same key, against four machines, produced one reservation and four refusals, including one from a transaction that could not lock anything and therefore never reached the insert and never reached the replay branch, while a reservation with its key already existed. Asking again immediately before refusing fixes it, and it works because each statement takes a fresh snapshot at our isolation level, so a sibling that just committed is visible.

The guarantee that survives all of this is narrow and worth stating exactly: five requests with the same key produce one reservation and four "somebody was faster". The losers do not see the winner yet, because it had not committed when they looked. There is no version of that which does not involve waiting for each other, and waiting turns every busy minute into a queue.

The reservation becomes the binding

When the order completes, the machine has to go from reserved to sold. The obvious implementation releases the reservation and then writes the service binding, and it contains a window of a few milliseconds in which the machine is free. On a busy day that window gets found.

So the conversion writes the binding and flips the reservation to converted in one transaction, and the row stays held until the binding exists. At every instant of that transaction, a second order sees either the reservation or the binding, and both exclude it. There is no free moment because no statement ever creates one. It is the same shape as promoting a rack hold into a placement, one layer up.

The sweep is a cleaner, not a guard

A reservation expires after fifteen minutes by default, and a worker sweeps expired ones every minute.

The number is not the four hours a rack hold gets, and the difference is what the two are waiting for. A rack hold waits for a person with a cart: somebody has to walk to the hall, fit rails, move cables. A reservation waits for a checkout: payment screen, fraud check, confirmation. Meanwhile the most expensive inventory a hoster owns is unsellable. Fifteen minutes is generous for an interrupted payment somebody resumes, and short enough that a storm of abandoned orders does not empty the shelf for half a day.

The important part is where the deadline is enforced. It is checked inside the conversion, not by the sweep. If the sweep were the only place, a reservation that expired between two runs would still convert, and a guard that comes past once a minute guards nothing. The sweep makes the administration correct and returns the machine to stock; that is a cleaning job.

It also has no feature gate, which is a deliberate exception. There is a rollout flag over the whole datacenter module, and repeating it here would mean that switching the module off leaves machines reserved forever for orders nobody can see any more. A switch whose side effect is disabling a timeout is not a switch, it is a leak.

What we would write down

1. 01Put the arbiter in the database. The application should produce the readable refusal; the index should decide the winner.
2. 02Lock one candidate at a time. Locking a set holds rows you will not use, and the symptom is missing sales rather than an error.
3. 03Cap the scan. Per candidate locking trades a hostage pool for a long loop, and the loop needs a limit too.
4. 04A unique violation aborts the transaction. Retrying without a savepoint produces an error that names the wrong thing.
5. 05Look at the constraint name. One table often has several unique constraints, and they usually mean different things to the user.
6. 06"Nothing available" and "somebody was faster" are different messages. Building the candidate list twice is how you tell them apart.
7. 07Promote, do not release and rewrite. Any sequence with a free instant in it will eventually be found.
8. 08Enforce the deadline at the point of use. A periodic sweep is housekeeping, and housekeeping is not a rule.

For clarity about what this is: the datacenter logistics work sits behind a rollout flag that is off by default with a per organisation setting on top of it, so none of this is purchasable today. Two things are unproven. The expiry sweep was measured as a function rather than as a running worker. And a raw request against the server actions with the gate closed was not attempted.

From the platform work on Provibr. The same promote instead of rewrite pattern applied to rack units is in the hold becomes the placement, and why free units in a rack are not the same thing as sellable capacity is in rack space is not capacity.