How an agent picks the right edge server

If you run an agent on your customers' hardware, the address it dials lives in two awkward places: in public DNS, and in whatever credential file you handed out at enrolment. Both are hard to change afterwards, and one of them tells the entire internet where your control plane is.

We put a layer of edge servers in front of ours. This is what that took, including the parts that only showed up once real agents ran through it.

The problem is the address, not the traffic

The gateway hostname resolved to the origin. Every enrolled agent knew it, and so did anyone who could type dig. A volumetric flood aimed at that address takes down the control plane for everybody, and there is nothing clever to do after the fact.

The goal was narrow, and stating it keeps the design honest: the origin address disappears from the path agents use, so a flood hits a machine that can be replaced in ten minutes. It was explicitly not a goal for the edges to protect the content. Mutual TLS already does that end to end, and an edge that could inspect the traffic would be a downgrade.

Layer four, because terminating TLS would break the thing that matters

The obvious design is a reverse proxy that terminates TLS at the edge. It is the wrong one here, for two reasons specific to this kind of system.

The gateway authenticates agents on their client certificate, pinned against a fingerprint stored per agent. Terminating TLS at the edge ends the mutual handshake there, so the origin would have to trust whatever the edge asserts. And it would mean copying the gateway's private key onto several cheap virtual machines whose whole purpose is to be disposable.

So each edge is a plain TCP forwarder. It sees ciphertext, holds no secrets and runs no application, which is a small enough surface that "replace it" answers most problems.

It costs one thing. Every connection now appears to originate from the edge, so the origin no longer knows who is calling.

PROXY protocol v2, and the difference between a claim and a measurement

The standard fix is PROXY protocol: the forwarder prefixes each connection with a small binary header carrying the original address. The gateway parses it and puts the real address back into its logs.

The design decision worth copying is to keep two values rather than one.

• The address from the header is what the other end says the client is. It is a claim.
• The outer socket address, the machine the bytes actually came from, is something we measured ourselves.

A forged header can make the first one lie. The second one is still true. Keeping them as separate fields means a log line can be read years later without wondering which of the two it is, and it makes the honest answer to "can we trust the IP" easy: nothing hangs on it except a log line, because identity hangs on the peer certificate.

Two implementation notes that saved us grief:

Sniff, do not require. A migration to PROXY protocol is not atomic. If the first bytes look like a TLS ClientHello rather than the protocol signature, push them back onto the stream and forward the connection untouched. That makes deploying the parser and enabling it on the forwarders two independent steps.

Keep a map keyed on the inner connection. Once the bytes reach a gRPC library the peer address is a loopback port, and both the agent and the edge have become unfindable. Keying the real peer on the source port of the internal connection, and clearing it on close, is the only place that information survives.

There is a small operational win hiding in there too: a connection that opens and closes without sending anything produces no log line at all. Health checks do that constantly, and not writing them removed tens of thousands of useless lines per day.

Discovery: the A set is the list

How does the agent learn that there is more than one edge? The options were TXT records with a structured payload, SRV records, or simply several A records on the same name.

Multiple A records won on one argument. The A set already is the list of places to connect. A TXT record would be a second source of truth for the same fact, and the two can disagree. SRV would add weights we do not need, because the agent measures for itself rather than being told which one to prefer. The TTL is short, so removing an edge is a DNS change plus one TTL of patience.

What a client actually does with several A records

Before building any selection logic, it was worth measuring what happens for free. A modern HTTP client tries the addresses in the record set and spreads its connect timeout across them. We put a known-dead address first and measured:

The pattern is clear enough to design against: hard rejections are free, and a black hole costs roughly the connect timeout divided by the number of addresses. Failover therefore already worked without any agent change, so the DNS switch could ship before the selection logic did. It also means your connect timeout is now a function of how many records you publish, which is not obvious until you have this table in front of you.

Failover is not selection

Free failover picks whichever address answers first, which is not the same as the best one. An agent in Amsterdam sat on an edge in another country for days, because that one answered first and the session then lived for a week.

So the agent measures. The details that mattered:

Probe with a full TLS handshake, not a TCP connect. An edge can accept a connection and forward nothing, which is exactly the failure the probe has to catch. A bare connect is answered just as cheerfully by a broken forwarder as by a working one.

Give each probe its own budget, in parallel. An edge that accepts and then goes silent can leave a handshake hanging past the connect timeout, so the per-address budget sits alongside it rather than relying on it.

Persist the whole ranking, not the winner. The chosen address and the ordered list go into the agent's state file, together with the address set they were measured from. That makes a restart cheap, and it means a failover survives a reboot instead of cheerfully returning to the address that just failed.

Invalidate on the set, not on the choice. If an edge is added or removed, the measurement is stale even if the chosen address is still in it.

Do nothing when there is nothing to choose. With fewer than two candidates the whole mechanism is skipped, which is what happens in development where the name resolves to one address.

And the rule that governs all of it: selection must never become a new failure mode. DNS refuses, nothing answers, the state file cannot be written: every one of those paths ends in "no choice", and the agent connects exactly as it did before any of this existed.

Coming back is harder than failing over

Failing over is easy. The hard half is that afterwards the agent stays on the slower edge for as long as that session lives, which can be weeks.

So a watch runs alongside the live session and does two things and nothing else. It tries the addresses ranked above the current one, using the same full handshake, and it re-measures everything when the last measurement is old. The numbers, and the reason for each:

• Interval of five minutes with full jitter, so probes from a fleet do not synchronise. Every minute costs five times as much and wins nothing, because the hysteresis below makes any return take two rounds anyway.
• A streak of two before switching back. One successful probe is an edge that just finished restarting.
• Two thresholds for the periodic re-measure: an absolute one of 20 ms and a relative one of 25 %. Both have to be met. The absolute one alone would approve a move from 100 ms to 79 ms; the relative one alone would approve 4 ms to 3 ms. Two edges in the same data centre differ by less than a millisecond, and without this they would swap places every other day.

Switching is a clean reconnect with its own reason, sitting next to the other planned session endings rather than being modelled as an error. That matters more than it sounds: an error would feed the exponential backoff ladder, so an agent that returns to its preferred edge would be punished with a growing delay for doing the right thing. The message outbox replays and the deduplication store makes sure nothing is executed twice.

Watching it from the other side

The platform watches the same thing from its end, and two details there are worth stealing.

Resolve the name the way the agent does, through the system resolver rather than a direct DNS query. A direct query failed outright on a machine whose resolver pointed at itself, and more importantly the system resolver is what the real traffic uses, so it returns the same list.

Take two numbers from one socket: the TCP connect, which is the round trip to the edge itself, and the full TLS handshake, which has to travel through the tunnel to the origin and back. Health hangs on the second one. A forwarder with a dead backend answers the first one perfectly.

The last lesson is a firewall one. Docker bypasses the host firewall entirely. The rules that only edges may reach the gateway ports have to live in the chain Docker itself consults, or they are decorative while looking active. And that chain sees traffic in both directions, so our own outbound probes from the origin to the edges matched the same destination port and were dropped, making the monitor report every edge as down while every agent was fine.

What it does not solve

The edges do not protect the content, and it would be a mistake to start treating them as a security boundary. An address that was down during a full measurement is only rediscovered by the next one, a known and accepted gap. And the DNS TTL is a floor on how quickly an edge leaves rotation, which is the price of letting the A set be the list.

This is how the agent reaches the platform at Provibr, where it connects outbound only and keeps credentials on the customer's own host.