Planning IPv4 address pools for a VPS product

Everything else in a VPS product is elastic. Need more CPU, buy a node. Need more disk, buy disk. Need more addresses and you are on a waiting list, paying a broker, or filling in a justification form for a regional registry.

IPv4 is the one input that behaves like a physical good, and it usually gets the least design attention, because at the start you have a /24 and 254 feels like plenty. It stops feeling like plenty at exactly the point where redesigning it is expensive.

Start with what a block really gives you

The usable count is never the size of the block. Two addresses go to the network and the broadcast, and one more goes to the gateway. That is not a rounding error at small sizes, it is most of the block:

A /29 sounds like eight addresses right up until you sell five of them. If you buy address space in small blocks, budget for the overhead as a line item; at /29 you are throwing away three eighths of what you paid for.

Two boundaries are worth encoding in whatever you build. We accept a /8 at the largest, because for anything bigger a usage bar stops meaning anything, and a /30 at the smallest, because /31 and /32 have zero usable addresses under normal host rules and accepting them just produces a pool that can never allocate. Refusing them explicitly is better than accepting them and being empty.

One more subtlety: the gateway address is allowed to sit inside your allocation range and is simply skipped. That is deliberate, because it is the easiest arrangement for a customer to reason about, but it means the capacity of a pool is a calculation and never a stored number. Anything that keeps a counter of free addresses will disagree with reality on the day somebody moves a gateway.

Shape pools around what you sell, not around what you bought

The unit a customer sees is a plan. The unit you bought from a broker is a block. Those two are almost never the same shape, and tying them together goes wrong the first time you buy a second block.

The arrangement that has held up: a plan points at a pool, and a pool is either one prefix or a named group of prefixes. Adding capacity to a product is then adding a prefix to a group, which is one row. The allocator walks the group in order and takes the first address it can get, so a group is also a drain order: put the block you want consumed first at the front.

Sub-blocks, and the mistake that silently double-allocates

Customers do not use a /24 as one flat network. They cut it up: a piece behind one router, a piece on a different VLAN, a piece with its own gateway. So an address plan needs child blocks.

This is where the interesting failure lives, and it is worth stating precisely because it fails silently.

The mechanism that stops two simultaneous orders from getting the same address is a partial unique index on (prefix, address) limited to rows that have not been released. That index is per prefix. If two prefix rows can each hand out the same address, then as far as the index is concerned those are two different addresses, and the platform will hand the same IP to two machines with nothing whatsoever going wrong in the software.

The index cannot fix that. The layout has to. So the address space is a strict partition, and the rules that make it one are:

1. 01A child lies entirely inside its parent and is strictly smaller.
2. 02Siblings may not overlap.
3. 03The parent hands out nothing that falls inside a child block. The whole block is carved out of the parent's window, not merely the child's own allocation range, because that would make the answer depend on a range some administrator moves next week.
4. 04Nesting is one level deep.

With those four, "which prefix does 192.0.2.70 belong to" always has exactly one answer, and the existing index is sufficient.

What a child inherits is a separate set of decisions, and the answers are not uniform. Gateway: never (a child without its own gateway has no reason to exist). DNS: yes, if its own field is empty, because resolvers belong to the network rather than to the block. Allocation range: only where the parent's range actually touches the child, since the parent's range is expressed in the parent's addresses and usually does not overlap the child at all; inheriting it blindly produces an empty window.

Let the database referee

The temptation, when two orders arrive at once, is to lock something or to keep a counter. Both are worse than the boring answer.

Both orders compute the same "lowest free address" and both try to insert. One insert wins, the other hits the unique violation, and the loser moves to the next candidate. The database is the only party that sees every writer, including the one in another process, another worker or another browser tab.

The corollary matters when you write the code: do not check first. Reading "is this address free" and then inserting opens precisely the window the constraint exists to close. The failed insert is the answer to "was it free", and it is the only reliable one.

One implementation detail that catches people: many ORMs wrap driver errors in their own error type, so the SQLSTATE you want is not on the exception you catch but on its cause. Check the whole chain, and use the constraint name to decide which message to show. A single table often has four different uniqueness rules, and making them all say the same thing sends the customer to the wrong field.

What happens when a pool runs dry

Three different failures hide behind "no address available", and lumping them together produces a support ticket every time:

The middle one is worth the extra code. A single prefix with unreadable data should not block the other prefixes in its group; note it, try the rest, and only report it if nothing at all worked. Otherwise one bad row takes down ordering for a whole product line.

And the exhaustion message should name the prefix and its CIDR. "No IP available" sends somebody to read logs. "Pool full: customer-block-2 (198.51.100.0/28)" sends them to buy addresses.

Give addresses back, and be careful about when

Release is the half that quietly leaks capacity.

Two rules that turned out to matter more than expected. First, releasing is not deleting the row: it sets a release timestamp. The partial unique index only looks at live rows, so the address is immediately reusable, while "who had 192.0.2.10 last year" stays answerable. That is most of the reason to run address management at all, and it is the question an abuse report asks.

Second, the release happens when the machine is confirmed gone, not when the delete was requested. Only the thing standing next to the hypervisor knows the VM actually disappeared; releasing on the outbound command hands the address to the next customer while the old machine is still running and still answering on it. The mirror case is an order that fails halfway: there the allocation is discarded rather than released, because it was never configured on anything.

There is a deliberate asymmetry here worth flagging, because we make the opposite choice one layer up. Addresses are reused, lowest free first. Machine IDs are never reused. A late status message about a deleted VM landing on a new row with the same ID corrupts two machines' histories; an address does not carry history in that way, and never reusing one would permanently exhaust a /24 after 254 machines. Same product, opposite rules, different reasons.

Reservations belong in the same table as allocations

You will need to hold addresses back: a load balancer, a gateway on a block you have not deployed yet, the five addresses somebody promised a customer next quarter.

Put them in the same table as real allocations. Every reader that asks "what is taken", the allocator, the usage bar, the suggestion list in an address field, then counts them automatically, and there is no second administration to fall out of step. A separate reservations table is a second source of truth that will disagree with the first one within a month.

With one carve-out that is easy to get wrong: a reserved address must never end up in a machine's firewall allowlist. That list is built from the allocations belonging to one server, and a reservation belongs to no server. Get that wrong and one customer's VM starts accepting traffic addressed to somebody else's load balancer.

And IPv6

We recognise IPv6 input and refuse it politely rather than accepting it and computing something plausible: a half-working IPv6 path is more dangerous than none, because the numbers it produces look correct. The cheap decision that pays for itself is storing addresses, gateways and DNS servers as text rather than as a 32-bit integer, so adding IPv6 later is code and not a migration of every row you own.

The short version

• Budget the overhead. A /29 is five sellable addresses, not eight.
• Point plans at pools, and pools at prefixes. Buying more space should be one row.
• If you allow sub-blocks, make the address space a strict partition, or the uniqueness rule that protects you is quietly per-prefix and protects nothing.
• Let the unique constraint decide the race. Never check first.
• Release on confirmation, not on request, and keep the history.
• Name the pool in the exhaustion message. It is the difference between a ticket and a purchase order.

This came out of building address management into Provibr, where addresses are issued when an order is placed and handed back when the agent confirms the machine is actually gone.