A probe that cannot be talked into anything

Checking whether a customer machine is reachable is only meaningful from somewhere the machine is not. An agent inside the network can tell you the service is listening; it cannot tell you the route to it is broken. So you put machines of your own on the public internet and let them make the connection.

That is a feature and a liability in one breath. A probe is hardware we own, in a datacenter we share with strangers, holding an open control connection to our platform. If anything on that connection can persuade it to do something other than measure, it becomes the most useful foothold in the product.

One binary, and the token decides who you are

A probe is not a separate program. It is the agent our customers run, started with probe enroll or probe run instead of run, on the same monitoring engine and the same session RPC. A stripped down build cannot execute a hypervisor command at all, which sounds safer, but it means a second implementation of retry, backoff, certificate rotation and the check engine, and the second one gets less attention every month. We kept one binary and moved the boundary somewhere testable.

The obvious way to let one binary enrol as two things is then a field in the enrolment request saying which. It is the wrong way, and it is worth being blunt about why: a client that declares its own kind declares its own privileges, and every later check rests on a string the thing being checked supplied.

So there is no such field. There are two token tables: enrollment_tokens with a not null foreign key to agents, and monitoring_probe_enrollment_tokens with one to monitoring_probes. A token exists in exactly one of them, so what gets enrolled follows from something we issued rather than something the caller said. Both failures return the identical rejection, because differing messages would tell an attacker which sort of token they had found.

The certificate that comes back carries a common name of the form probe:<id> and, as its subject alternative name, a URI with no organisation in it, where the URI on an agent's certificate names the customer. That is for an operator reading a certificate and it is explicitly not the boundary: a common name is a string a certificate authority writes. The boundary is that a fingerprint resolves in agents or in monitoring_probes, never in both, because those are two tables.

A verdict for every message, not a list of the allowed ones

The session protocol is two envelope messages, each holding one of a set of alternatives. An allow list naming only what a probe may send stays green forever, including on the day somebody adds a fifth message type, and on that day there is a message type nothing mentions and therefore nothing refuses. So the tables in the gateway name every field of both envelopes with an explicit verdict: twenty two entries upstream, of which three real messages plus a sequence number are accepted, and fourteen downstream, of which one real instruction plus three session acknowledgements are accepted. A test reads the generated TypeScript and fails if a field has no verdict, and fails the other way too when a name in the table no longer exists in the protocol, because a rule that constrains nothing reads exactly like coverage. Adding a session message now costs one line, and that line is a decision with somebody's name on the commit.

Upward, a probe may send three things: hello, heartbeat and a check result. The denial worth explaining is command results: dispatch matches a result to a command by identifier rather than by sender, so a probe that could send one could close out a command belonging to any agent in the fleet. Downward, ten entries are refused, and four of those are what the whole design is for:

The registry a probe is not in

The table above is the provable half of the promise. The load bearing half is that there is no route. Command dispatch finds its destination by agent identifier; probes live in their own registry, keyed by probe identifier, and have no agent identifier at all. No lookup can return a probe session to the code that sends a credential push, even if somebody added a new kind of instruction and forgot the table entirely.

The single function that writes to a probe checks the verdict at the source rather than at the receiver, because a message that slipped through would arrive at a machine of ours running the same binary as a customer agent, which might simply carry it out. It also performs a check the table cannot express: command is accepted, but a command request is itself a choice of twenty four alternatives, all but three of them a family of provider instructions, so without it "a probe may receive commands" would quietly mean "a probe may receive a hypervisor command". The payload has to be exactly one field, and that field has to be the monitoring check. The test walks every payload field in the generated code and asserts that all except one are refused.

Choosing a probe, and failing over to the next one

Each location can hold more than one probe, so something has to decide. A hash modulo the number of probes is wrong on the day that count changes: almost every rule moves at once, and latency graphs all over the product jump on the same day for no visible reason. Rendezvous hashing moves only what was attached to the probe that left. The key is the rule revision and the location. The revision is immutable, so a rule the customer edits may legitimately land somewhere else, while the choice within one revision never drifts. The slot is deliberately not in the key: with the tick number in there, consecutive measurements would land on different probes, and a difference between two points could just as easily be a difference between two machines. Comparability over time is worth more than spreading load.

A probe can let you down in two ways, noticed at different times: it can be gone when you try to send, or it can accept the command and never answer, which surfaces only when the check's own timeout has passed. Both produce one more attempt inside the same tick, and never a second tick. A failover is not a second measurement, and if it were, an unlucky location would silently double the sample count. The ceiling is two physical attempts, because together they have to fit inside the tick deadline.

When a location has no healthy probe at all, the slot is still claimed and the source is recorded as unavailable. Never as down: a location of ours being absent says nothing about the customer's machine, and calling it an outage would be a false alarm generated by our own infrastructure.

Three places that refuse a private address

A monitoring rule is a field where somebody names a destination and a machine of ours connects to it every minute. Without limits that is a request forgery engine with a billing address on it. The target is an address allocation belonging to the customer's own server rather than free text, which removes most of the problem; the rest is a list of ranges a hosted probe may not touch, on top of the ones closed everywhere:

Mapped IPv4 forms are unwrapped first, or every rule above has a spelling that walks past it.

The check runs in three places and all three are needed. The platform refuses before sending, so a bad target never reaches a machine that could carry it out. The probe refuses at execution, because only there is it known what a name actually resolved to, and that can differ from run to run. And the platform reads the reported address once more when the result arrives, turning a probe that measured somewhere else into an invalid response rather than a data point. A private destination is a configuration error and never a failure: nothing was measured, so there is nothing to say about the target.

What we would write down

1. 01Never let the client name its own kind. Derive it from something you issued, and keep both rejections identical so the error is not an oracle.
2. 02Enumerate the whole protocol, not the permitted part. Let a test read the generated code and demand a verdict per field, in both directions, so a new message type is a red test rather than a gap.
3. 03Put the real boundary where it cannot be forgotten. A separate registry with no agent identifier means there is no route, whatever the table says.
4. 04Check the payload as well as the envelope. "Commands are allowed" is not the same statement as "this command is allowed".
5. 05Keep the assignment key free of the clock, and make a failover an extra attempt rather than an extra sample, so a change in the data can never be mistaken for a change in the instrument.

For clarity: hosted monitoring sits behind a rollout flag that is off by default, there is no entitlement layer under it, and none of this is purchasable today. What is unproven deserves naming too. The probe subcommand has never run end to end, because one of its network components does not compile on the machine this was written on; the location policies have been exercised only in their simplest form; IPv6 has not been measured anywhere. The wire itself was driven for real, from enrolment through a session to a check that came back up, and a forbidden message on that connection was refused.

From the monitoring work on Provibr. Why a hostname needs to be proven before it can be a target is in proving you own a hostname, and why an incident does not close just because the measurements stopped is in an incident that never closes on silence.