Blog ·
A backup that was never restored is not a backup
Three different proofs get called one word. What a green Proxmox Backup Server verify really says, what it cannot say, and why empty has to mean empty.
By Provibr
Everybody agrees with the title. Almost nobody's dashboard reflects it.
The reason is not laziness. It is that three genuinely different pieces of evidence arrive through the same pipe, get stored in adjacent columns, and end up rendered as one green tick. Once they are one tick, the strongest claim in the group silently borrows the credibility of the cheapest one, and a customer reads "protected" when what actually happened was that a job exited zero.
We spent a round pulling those three apart in our own schema, and a second round measuring a real Proxmox Backup Server to find out which of them it can actually give us. This is what we ended up with.
Three proofs, and they are not interchangeable
| Proof | What it means | Who can produce it |
|---|---|---|
| Provider completion | A job ran and a manifest was written | The hypervisor, at backup time |
| Storage verification | The storage re-read the data and it matched | The backup server, on its own schedule |
| Restore test | A selected restore point was brought back somewhere isolated and checked | Nobody, unless you build it |
The first one is the cheapest and the one every product shows. It is worth exactly what it says: a process finished. The guarantees behind each vzdump mode are about what that process could and could not promise about the state of the guest, and none of those guarantees survive contact with a corrupt chunk store afterwards.
The second is real evidence and it is easy to over-read. More on that below.
The third is the only one that answers the question a customer is actually asking, and it is the only one that requires you to build something rather than read something. That asymmetry is why it is usually missing.
What a green verify actually asserts
Proxmox Backup Server can verify a snapshot: it re-reads the chunks and checks them against the manifest. When that has happened, the snapshot listing carries a verification object with a state in it.
That is a genuinely useful thing to know, and it is narrower than it feels. It says the bytes on that datastore are the bytes that were written. It does not say the machine will boot. It does not say the application inside it will start. It does not say the database was in a consistent state at the moment of the snapshot. And if the backup is client-side encrypted, it does not say anybody still has the key: an integrity check on ciphertext passes perfectly well when the key is gone.
So the state lives in its own column in our schema, and it is documented in the code as belonging to proof two, never to be mixed with "restore proven". Proof three will get its own table on the day somebody builds it. Adding a fourth value to an existing enum would have been half a day's work and a permanent source of confusion.
A missing field is not a failure
Our column has three values: ok, failed and unknown. The middle one is not the interesting one.
Measured against a live Proxmox Backup Server 4.2 with real snapshots on it: a snapshot that has never been through a verify task simply has no verification key. The field is absent. That is a different thing from a verify that ran and failed, and collapsing them costs you in both directions. Default it to ok and every unchecked snapshot becomes a checked one. Default it to failed and every historical snapshot in your database is suddenly on fire. The third value exists to say the only true thing: nothing is known.
A plain Proxmox storage never fills the column in at all, because its content listing carries no verification concept. That is not a gap in our integration. That is the answer.
One more honest detail, because it is the kind that gets rounded off. The verification object we measured carries a state and a task identifier, and no timestamp. So the "verified at" moment we record is the moment we first saw that state, which is a different question from "when did the storage verify this". It answers "since when do we know this". If a future version exposes the real time, the column should start carrying it and the comment explaining all this should be deleted.
Empty has to mean empty
Here is the sharpest rule in the whole area, and it has nothing to do with verification.
Any system that lists restore points periodically ends up deriving something from a snapshot's absence: it was here last week, it is not here now, therefore it was pruned or deleted. That inference is load-bearing, and it is only valid if a successful empty listing is genuinely empty.
So we measured what a backup server actually answers when a listing goes wrong, because "an empty array" and "an error that looks like an empty array" have to be distinguishable. Four cases, all measured against that same live 4.2 instance:
- an empty namespace answers 200 with an empty array;
- a namespace that does not exist answers 400 with a not-found error, not an empty list;
- a datastore the token cannot audit answers 403;
- a bad token answers 401.
Three of those four have to become specific errors rather than results. If any of them is allowed to reach your reconciliation logic as "the listing succeeded and there is nothing there", then one typo in a namespace name marks a customer's entire backup history as gone. The failure mode is not an outage. It is a screen that quietly stops mentioning that backups exist.
This is the same class of trap as a hypervisor answering 200 before the work has started: the response code is fine, and it is not the answer to your question.
The credential should not be able to break anything
The most useful security decision here is also the most boring one.
The token a customer creates for a backup target needs to read snapshots and datastore status. It does not need to delete, forget, prune or verify. Those are separate operations that belong to whoever owns the storage, on their own schedule, with their own identity.
We built the integration with no writing command at all, which makes the claim structural rather than aspirational, and then checked it from the other side: with a token holding audit rights only, a delete on a snapshot answers 403. That is the shape to aim for. A read-only integration whose credential is capable of pruning is a read-only integration exactly until somebody writes a loop with an off-by-one in it.
There is a corollary that hosting providers get wrong in the other direction. If your platform deletes a server, the retention question is a decision the customer has to make, not a default your code picks. Removing the only remaining copy of somebody's data as a side effect of tidying up a row is a thing that has happened to real companies.
What a restore test has to be, if you build one
Proof three is where the work is, and it is worth writing down the constraints before writing the code, because most of them are about not causing an incident with your test.
A restore test needs somewhere isolated to restore into: a network with no route to production, temporary identifiers, temporary addresses, temporary hardware addresses. Never the production address, and never the production hardware address, because a duplicate on a live network is an outage you caused while checking that you could recover from an outage. It needs strict quotas, because a restore of a fleet's largest machine into a test pool is a capacity event. It needs a deadline and a guaranteed cleanup, since the machine will sometimes fail to come up and something has to remove it anyway. And it needs to not send notifications, because a monitoring alert from a test copy pretending to be the real system will wake somebody up.
It also needs an explicit completion policy, and this is where the whole thing earns its name. Booted is one level. Reachable is another. Guest agent responding is another. Filesystems mounted is another. An application answering a customer-defined health check is the level that matches what a customer means by "recoverable". Pick one, record which one you picked next to the result, and never let a lower level be displayed with the label of a higher one.
Only then does a server get to show "restore proven", and it should carry the date, the exact restore point, the environment it ran in, the checks that passed and how long the evidence is kept. Anything less and the phrase is decoration.
What we would write down
- Name the three proofs separately in your schema, before you have all three. Ours are separate columns and a table that does not exist yet, which is easier to read than one column with an apologetic comment.
- Store "unknown" as a real value. Any default that is not "unknown" is a lie about at least one of your rows.
- Make an empty listing prove it is empty, and turn every other answer into a distinct error before it reaches anything that reconciles.
- Give the integration a credential that cannot delete, and then verify from the outside that it cannot.
- Write down what "restore" would have to mean for you, at what completion level, before anybody puts a green tick next to it.
Our own state, said plainly: the first two proofs are built and measured, and we have no restore test. The backup feature sits behind a flag and is not for sale. That is the difference between a plan and a claim, and it is the same difference this whole article is about.
The integration behind these measurements was built for Provibr. The related timing question, what happens when a backup takes longer than the deadline you gave it, is in a timeout is not a stop button; the container side of Proxmox backups is in VM or container on Proxmox.
