How a signed agent update refuses itself

Any piece of software that updates itself is remote code execution that you invited in. That is not an argument against auto-updates, which are the only realistic way to keep a fleet of agents on customer hardware from rotting. It is an argument for being precise about what the update mechanism promises, because most of them promise less than their documentation implies.

We run an agent on other people's servers. It talks to our platform, it applies commands to their hypervisor, and it replaces its own binary when we ship a new version. This is what we had to build before we were willing to let it do that.

A checksum answers the wrong question

Almost every self-updating tool publishes a SHA-256 next to its download, and a lot of them stop there. A checksum is useful, and it does not do the job people think it does.

The middle row is the whole problem. The checksum and the artefact usually live in the same place and are usually writable by the same process. If an attacker can replace one, they can replace the other, and the verification will pass with enthusiasm.

So the checksum is a cheap pre-flight test and explicitly not the trust boundary. It catches a truncated download in one pass over bytes we were streaming anyway. The signature is the anchor.

Where the public key has to live

Here is the first decision that determines whether any of this is real: the verifying key cannot be a file that ships alongside the binary, because then compromising the delivery path compromises the key too.

Our agent gets the platform's Ed25519 public key compiled into it, from a build-time environment variable. In a release build there is no other source. In a debug build a developer can point it at a local key with an environment variable, and that path is compiled out entirely by a cfg attribute, because a key that can be overridden at runtime is not an anchor but a suggestion.

Two details around that matter more than they look.

The build script declares that it must re-run when that environment variable changes. Without that line, a release build sharing a target cache with an earlier build happily reuses the key from the previous compile, and nobody notices until it reaches a customer.

And the key parser is strict about the encoding rather than the length. An Ed25519 public key in SPKI form is a fixed 12-byte prefix followed by 32 bytes; an X25519 key is also 32 bytes, with a different algorithm identifier. Accepting anything 32 bytes long would take a key for a different algorithm and fail later, somewhere that makes no sense.

Refuse before you download

When the platform tells the agent to update, the agent runs through the whole list of preconditions before it fetches a single byte:

1. 01Is there a usable SHA-256 in the directive?
2. 02Is there a download URL?
3. 03Is there a signature, and is it exactly 64 bytes?
4. 04Are we already running this version?
5. 05Does the privileged apply script exist on this host?
6. 06Does the local trust anchor file exist?
7. 07Does this binary have a platform key compiled in at all?

Step three deserves the attention. An older platform that does not populate the signature field should fail here, loudly, rather than quietly falling back to the weaker check we used to do. Optional verification is not verification. Making the missing field a hard error is the difference between a mechanism and a habit.

The order of the last three is deliberate too. The "your host is missing the privileged step, run the installer again" messages are more actionable than "this binary has no platform key", so they come first.

Sign the bytes, not a description of them

The signature covers the artefact exactly as it lands on disk. Pure Ed25519, no pre-hashing, no canonical envelope, no metadata included. The reason is not purity. It is that two completely different implementations have to reach the same verdict.

On the agent it is ed25519-dalek with verify_strict, which refuses signatures involving torsion components. With one key and one file there should be exactly one valid signature, and the non-strict variant does not guarantee that.

On the host, the privileged script that actually installs the update verifies the same bytes with openssl pkeyutl -verify -pubin -rawin. Same key, same bytes, same answer. If we had pre-hashed, or signed a manifest instead of the artefact, the shell-level check would have to reimplement our framing, and the two checks would drift.

That is also the check we run by hand on the server after every release, against the production public key, before believing that a release happened.

The manifest next to the artefact carries the version, build identifier, checksum, size, release timestamp, artefact kind and the signature. It is metadata for the panel, and none of it is trusted: the signature in it is over the binary, not over the manifest.

The agent cannot install its own update

The agent runs as an unprivileged user. It cannot write to the directory its own binary lives in, and it should not be able to.

So it stages. It writes the signature and the checksum into a staging directory first, and only then renames the verified binary into place beside them, because the privileged script uses the presence of that binary as its start signal and the other two files have to be there already. Then it exits cleanly, and the service manager restarts it, running a privileged pre-start step first.

That script is worth describing, because its assumptions are the opposite of what you would guess.

It always exits zero. A failing pre-start step would prevent the service from starting at all, and a broken update that takes a customer offline is far worse than a broken update that quietly does not apply.

The staging directory is not trusted. Everything in it was written by the unprivileged agent user: the binary, the checksum, the signature. Anyone who takes over that user can replace all three. That is why the signature, verified with the key on the privileged side, is the only thing that decides.

It copies before it checks. The staged bytes are copied exactly once into a fresh root-owned temporary file next to the destination, and every check after that reads the copy. The file that gets verified is the exact inode that gets renamed into place, so there is no window in which a different file is swapped in between the check and the install.

The rest is deliberately boring: refuse symlinks, require the checksum to be exactly 64 hex characters, require the file to start with the ELF magic number, widen permissions to executable only after the signature passes, start it once as the service user rather than as root, and only then rename it over the live binary. A signature says where bytes came from. It does not say they deserve root.

The release we signed with the wrong key

The mechanism above works. We still shipped a broken release, and the way it failed is the useful part.

Our build script took a flag for "this is a production release". At the time that flag governed publication only, not which key was used to sign. A release built without also naming the production key on the server was signed with the development key, had the development licence key compiled into it, and then verified locally against that same development key. The check said the artefact was fine, because it was internally consistent. It just was not ours.

Every real agent refused the update, correctly, because the signature did not match the key compiled into it. The mechanism did exactly what it was built for. The mistake was upstream of it.

Two guards came out of that, both in code rather than in a document. Building for production without naming the production key is now a hard failure that stops the script. And the signing key and the verifying key have to be on the same machine: checking a signature with a key fetched from somewhere else proves nothing at all.

The general lesson is worth stating plainly. A verification step handed both halves of a matched pair from the wrong environment will pass, and it will pass confidently. Signature schemes protect against attackers. They do not protect against you.

A licence is not a binary, and both are signed

We sign two different things with the same key: agent binaries and licence files. That is a hazard, because a signature over one should never be accepted as a signature over the other.

The separation is structural rather than a naming convention. A licence file signs a text envelope built from the format identifier, the salt, the nonce and the ciphertext, joined with newlines. A binary signs raw ELF bytes. The install script checks for the ELF magic number before it verifies anything, so a valid licence signature can never be presented as a valid binary.

The licence flow has one more property worth stealing: it verifies the signature before deriving the decryption key. Argon2id with a 64 MiB memory cost is expensive by design, and doing that work for a file that was never ours is a free denial of service. Cheap authenticity checks belong before expensive cryptography.

What this does not buy

Signing binds an artefact to a key. It says nothing about whether that key is still yours, and nothing about whether the version you signed is any good. If our signing key leaks, every agent accepts what the attacker builds, and rotating it means rebuilding every binary and replacing the trust anchor on every host, because the key is compiled in. That is the price of the key being an anchor rather than a suggestion, and it is the right trade, but it is a real cost and it belongs in the plan rather than in a footnote.

This is how the agent update works at Provibr, where the agent runs on the customer's own hardware, connects outbound only, and keeps hypervisor credentials on that host.