Where hosting panel credentials should live

Every hosted control panel for server automation asks you the same thing in its setup wizard: the address of your hypervisor, an API token, the root credentials of your web hosting panel. Then it stores them, because it needs them on every request.

That is a reasonable thing for the software to want and a genuinely uncomfortable thing to have agreed to. You have just handed a third party's database the ability to delete every machine you own. Not through a bug and not through malice: that is the normal, documented, working behaviour of the product.

It is worth being precise about what the alternatives actually buy you, because "we encrypt them at rest" is the answer everyone gives and it is close to meaningless on its own.

Encrypted at rest, decrypted by whom

If a SaaS stores your hypervisor token encrypted, the key has to be somewhere that same SaaS can reach, because it has to use the token on your behalf while you are asleep. Encryption at rest protects against a stolen disk and a leaked backup. It does not protect against the thing you are actually worried about, which is the platform itself being compromised.

That is not an argument against encrypting at rest. It is an argument that the question "where is the key" is the only question, and that a control plane which can decrypt is a control plane which can be made to decrypt.

The structural alternative is the third row: to not have the secret at all. Which sounds glib, until you look at what it costs to arrange.

Keep the secret where the machine is

The shape that works: a small agent runs inside the customer's own network, holds the credentials locally, and connects outbound to the platform. The platform sends instructions. The agent holds the keys.

That inverts the blast radius. A compromise of the control plane can now issue commands, which is bad and which we will come back to, but it cannot walk away with a file full of hypervisor tokens for every customer, because there is no such file.

The parts that make it real are unglamorous.

The vault, and what it honestly protects against

Our agent stores credentials in a local file encrypted with XChaCha20-Poly1305. The key is not the file on disk: it is derived with HKDF from a key file mixed with the machine's own identity, and the additional authenticated data binds the ciphertext to the format version and to the agent's ID.

The practical consequence is the one worth designing for. Copy the key file and the vault to another machine and it does not open. Restore last month's backup of the data directory onto a different host and it does not open. Every failure mode, wrong key, different agent, altered bytes, produces the same indistinguishable error, which is the point.

Here is the part that belongs in the documentation of every product that does this, and usually is not there. This does not protect against root on the host. It cannot, and it does not need to: the customer owns the host and owns the credentials. What the vault protects against is a leaked file, not the owner of the machine. A vendor who implies otherwise is selling you a feeling.

The credentials never touch the platform database

This is the part that is easy to get wrong while believing you got it right, because the code looks fine either way.

When somebody types a hypervisor token into our panel, the row that goes into the command table contains field names only. The values ride in a separate transient message that is not stored, and go straight into the delivery to the agent. The dispatcher never logs them and never persists them.

If you are evaluating a product on this, the question to ask is not "do you store credentials". It is "what is in your audit log, and what is in your queue". A queue that carries the secret is a database that carries the secret with extra steps.

Deleting a secret is a different problem from storing it

Removing an integration in the panel has to result in the credentials disappearing from a machine you do not control, which is a harder problem than it looks.

The obvious implementation is a reconciliation: ask the agent what it has, compare against what the database says it should have, delete the difference. That is shorter and it is dangerous, because the subtraction fails towards everything is gone. One failed read, one truncated list, and you have generated an instruction to wipe every working integration on that host, and those credentials are not recoverable without the customer typing them again.

So the instruction comes from positive knowledge only: a row exists because somebody genuinely deleted an integration. A failed read produces zero rows and therefore zero instructions, and the secret stays where it was, which is exactly the state you were in before.

It is delivered twice: immediately over the event bus, and again on every session handshake, for the agent that was offline, the message that was dropped, the gateway that restarted. Both paths read the same table, so there is no second truth and no second ordering.

What a stolen agent certificate can and cannot do

If the agent authenticates with a client certificate, the honest question is what happens when somebody takes it. Being specific about this is more useful than a security page.

It can impersonate that one agent. The session port requires mutual TLS, and on top of the TLS check the peer certificate's fingerprint is looked up in the agent table, which has a unique index. A fingerprint therefore points at exactly one agent, and the identity claimed in the handshake is compared against it. An attacker with the certificate becomes that agent, and can send it whatever an agent may send: status events, results, telemetry.

It cannot read other customers' data. Everything is scoped by the organisation that the agent belongs to, resolved from the certificate rather than from anything the client says about itself.

It does not by itself yield the credentials. The certificate is the identity used to talk to the platform. The vault key is separate, lives on the host, and is bound to that host's machine identity. Someone who has both has root on the machine, and at that point the certificate is the least of it.

It stops working when you say so, in about a minute. Revocation is a database state, not a CRL and not OCSP. A new connection is refused immediately. An existing connection is re-checked on a one-minute interval, so an already-connected attacker has at most that. If the status check itself keeps failing, the session is closed after three consecutive failures rather than being kept open indefinitely, which bounds the worst case at roughly three minutes.

And the agent cleans up after itself. On revocation it wipes the vault, the vault key, its identity, its session state and its queues, in that order, credentials first. Each step is best-effort and logged individually, because half a wipe beats no wipe.

Certificates are also short-lived, ninety days, renewed automatically at the halfway point with an hourly check. That is not a defence against theft so much as a bound on how long a theft nobody noticed stays useful.

What this does not fix

Three things, stated plainly, because a threat model with no residual risk is a marketing page.

A compromised control plane can still issue commands. It cannot steal your tokens, but it can tell your agent to delete a machine, and the agent will do it. Moving the secrets moves the blast radius; it does not remove it. What you get back is that the damage is bounded by what the API can express and is visible in an audit trail, rather than being an attacker with your credentials doing whatever they like directly against your hypervisor at three in the morning.

Root on the agent host reads everything. By design. If that host is compromised you have a compromised machine inside your own network, and it has your hypervisor credentials because that is where they need to be.

Deletion depends on the agent coming back. If a host never reconnects, the instruction to forget its credentials sits open. That is visible in the panel instead of being silently marked done, which is the correct trade, but it is a real limitation and worth knowing before you rely on it.

Questions worth asking a vendor

Whatever you end up buying, these five separate the products that thought about it from the ones that wrote a paragraph:

1. 01Where is the decryption key, physically? If the answer is "in our platform", encryption at rest is protecting against disk theft and nothing else.
2. 02What is in the audit log and in the job queue? Secrets leak into both far more often than into the table somebody was careful about.
3. 03What happens to credentials when I delete an integration, and can I see whether it actually happened?
4. 04How long does revoking access take, and does it apply to sessions that are already open?
5. 05Does anything need an inbound port? A firewall rule that lets a vendor's IP range into your hypervisor's management interface is a second, quieter version of the same problem.

This is how we built it at Provibr: the agent runs on the customer's own hardware, holds the credentials locally, and connects outbound only.