A secret that travels once

Some monitoring checks need credentials. You cannot ask a PostgreSQL server whether it is healthy without logging in, and a Redis instance with a password set answers nothing useful until you have sent it. So a monitoring rule has to carry a secret from us to whichever probe or agent happens to run the next attempt.

The cheap version of this is a column. Put the password in the rule, read it when you dispatch, send it. That version is wrong in a way that only becomes visible thirteen months later, and this is the shape we built instead.

The two routes that already existed, and why neither works

A monitoring command reaches a source through two places, and both are unsuitable.

The rule revision table is immutable by design, which is exactly right for a revision and exactly wrong for a credential: nobody can rotate a password that lives in a row that cannot change without producing a new revision of every rule that used it.

The command row is worse, because it is an audit row and it travels through the message bus on its way out. Our own project notes have said for a year that a command audit row contains field names and never values. A password that lands there once sits there for the entire retention period, readable by anybody with database access, and copied into whatever backup ran that night.

There is also a transient side channel for exactly this kind of thing, used when somebody orders a machine and a root password has to reach the agent without being stored. It does not fit here either. That channel exists for secrets the creator of a command already knows, and here the creator is a scheduler that runs minutes early and does not yet know which source will take the job.

So the four protocol messages added this round have no username field and no password field. It is not that the fields are unused; they do not exist.

Sealed at the moment of sending

The envelope is created in the session worker, immediately before the message is built, once per attempt. That is not tidiness. It is the only point in the system where all four of the things it has to bind are known at the same time: which source is on the other end of this session, which attempt this is, which revision of the rule it belongs to, and when it expires.

The consequence is that it cannot be anywhere else, and that is testable rather than assertable. A test plants a known password, runs the real path, and searches for it in the command payload, in the event rows, in the stored check configuration, in the message bus, in the raw protocol bytes both as UTF-8 and as base64, and in the gateway log. It requires zero hits in all of them. A list of claims about where a secret does not go is worth nothing; a search is worth something.

The plaintext exists for one expression. One function produces it, the next consumes it, and after that there is no reference. JavaScript cannot wipe a buffer, so the only option is to never keep one. On the other side, in Rust, it goes into a type that zeroes itself when dropped.

The construction

An ephemeral X25519 key pair per attempt, a Diffie-Hellman against the public key of exactly one source, HKDF-SHA256 to derive the encryption key, and AES-256-GCM over the plaintext. The wire carries the ephemeral public key and a nonce, ciphertext and tag.

The two public keys go into the HKDF salt, so the derived key is bound to that pair rather than to a shared secret that could in principle arise from another pair. Same idea as the key schedule of a standard hybrid encryption scheme, with the simplification that is honest here: there is a fresh ephemeral key per attempt, so no key is ever used twice.

Four fields go into the additional authenticated data:

The identity of the source is not in that list because it does not need to be: the sealing is done to one public key, so another source cannot open it at all.

Length prefixes, because a separator is ambiguous

The additional data is built with each field's length in front of it rather than with a delimiter between fields. This is the kind of detail that reads as pedantry until you write out the alternative.

With a:b as the format, the pair ("x", "y:z") and the pair ("x:y", "z") produce identical bytes. The binding this data is supposed to establish then does not distinguish two different situations, and an attacker who controls part of one field can shift the boundary. Four fields with a four byte length in front of each cannot have that problem. Both sides build it byte for byte the same way, with a shared test vector so that a divergence is a failing test rather than an envelope nobody can open.

Why not the cipher we already use

The agent's own credential vault uses XChaCha20-Poly1305, and using the same construction here would mean one answer to "how do we encrypt a secret" instead of two. We measured, and it does not work:

node v24.20.0   crypto.getCiphers() includes "chacha20-poly1305"
bun  1.4.0      crypto.getCiphers() returns 28 names, none of them chacha
                createCipheriv("chacha20-poly1305", ...) -> "Unknown cipher"

Production runs on Node, so it would work in production. But the test harness of this repository is bun, and an AEAD the harness cannot execute is an AEAD whose round trip is never proven. The only check on the envelope would then live in a test that gets skipped, which is the exact flavour of vacuously green measurement we spend a lot of effort avoiding.

The alternative was to write the missing piece by hand. That was actually done, and it worked against the published test vector, and then it was deleted. Cryptographic code that exists only to route around a test harness does not belong in a product repository.

So: AES-256-GCM, the same AEAD that already encrypts this secret at rest, present in both runtimes, and already in the agent's dependency tree. The 96 bit nonce that makes people reach for XChaCha in the first place is not a concern here, because the reason XChaCha exists is to let you pick nonces randomly under a long lived key. There is no long lived key. Every envelope has a fresh ephemeral one, so every AEAD key is used exactly once and a repeated nonce would not be a second message under the same key.

Where the source keeps its half

The private key sits next to the identity of the agent or probe, in a file of thirty two raw bytes with mode 0600, beside the client certificate's private key. Not in the credential vault, and the three reasons are worth listing because they were the actual decision.

Revoking wipes it for free: revocation already deletes that directory, and a key stored anywhere else would need its own cleanup rule, which the first round that forgets it leaves behind on a revoked host. It belongs to one enrollment, and the platform stores the public half next to the certificate, so a fresh enrollment should produce a fresh pair, which it does because the directory is empty. And the vault is not always available: it is unlocked with material derived from the license file, and a monitoring probe has no license and therefore no vault. Putting the key there would force a second mechanism for probes, and then there are two answers to one question.

It is deliberately not held in memory either, which is tempting for a probe that stores nothing. The platform keeps the public half, so after a restart it would no longer match and every envelope would be unopenable until an operator re-enrolled. A monitoring location that quietly stops doing authenticated checks after a reboot is precisely the kind of outage nobody notices.

No fallback, ever

If a source has no key, the check does not run. It reports a configuration error naming the broker as unavailable, and that is the whole behaviour. There is no plaintext fallback and, more importantly, no misleading down status for the target: an unmonitorable check must never look like a broken service, because that is an alert that sends somebody to the wrong machine at three in the morning.

What we would write down

1. 01An audit row is not a delivery channel. Anything written there is there for the whole retention period, in every backup.
2. 02Do not add the field. A protocol message without a password field cannot leak one through a code path somebody forgets to review.
3. 03Seal where the bindings are known. If the creator of a job does not yet know who will run it, the secret cannot be created with the job.
4. 04Prove absence by searching, not by asserting. Plant a value and hunt for it in every serialisation, including base64.
5. 05Length prefix your authenticated data. A separator makes two different tuples produce the same bytes.
6. 06Choose the primitive your test harness can execute. A stronger construction that is never exercised is weaker in practice than a good one that is.
7. 07A key that is an identity belongs with the identity. Then revocation cleans it up and re-enrollment replaces it, without a rule anybody has to remember.

For clarity about what this is: monitoring sits behind a rollout flag that is off by default, so none of this is purchasable today. Two things are unproven. The password based authentication exchange was measured only against the reference implementation of the protocol library and not against a real server. And the Rust probe binary was not exercised over the wire in this round.

From the platform work on Provibr. Why a probe treats every message it receives as untrusted is in a probe that cannot be talked into anything, and why the credentials of a customer's own panels never reach us at all is in where hosting panel credentials should live.