A timeout is not a stop button

Every remote command needs a deadline. That part is uncontroversial: a process that can wait forever will eventually wait forever, and one stuck request will hold a worker, a connection and a row in your queue until somebody restarts something.

What is much less obvious is what the deadline actually does. It ends the waiting on your side. It does not, in the general case, end the work on the other side. Everything difficult about long running operations lives in that gap.

We run an agent on customer hypervisors and send it commands. A backup of a virtual machine is the command that made us take the gap seriously, because a backup can honestly take an hour, and an hour is exactly where our own limits sit.

The cap is not a preference

Our agent clamps every command deadline to one hour. The platform may ask for more; it will not get it. The clamp is a single min() on the value that arrives in the request, with zero read as "not set" and a fallback of two minutes when the field is missing.

That clamp has a consequence people miss when they set timeouts in a config file. If the platform believes in a deadline the agent does not, you have two systems disagreeing about when a command is dead, and the disagreement shows up as the worst possible symptom: the agent gives up, reports, and moves on, while the platform is still holding the row open and waiting for an answer that will never come with the shape it expects.

So the rule we ended up with is dull and worth writing down anyway. Every timeout on the platform side is chosen with the agent's ceiling in the room. Our backup and restore deadlines are one hour precisely because that is the ceiling, and out of several dozen command deadlines they are the only two that touch it.

Deadlines are not a style choice

Here is what our command deadlines actually look like, and why each one is what it is.

Two of those rows are stories rather than numbers.

The cloud machine one used to fall through to the default of two minutes. Two minutes is enough for an image that happens to be current, and not enough for anything else, so the platform reported a failure while the installation inside the machine carried on and finished a few minutes later. The order got marked failed. The machine was running. That is the worst kind of wrong: not an error, a contradiction, and the customer is the one who has to resolve it.

The migration one is the longest deadline that is not the agent's ceiling, and the reasoning behind it is a count rather than a feeling. The migrator makes one API call per user and one per server against the destination panel. At a thousand servers that is a thousand calls out and a thousand answers back, with a full read of both sides before it and a recount after it.

The costs of being wrong are not symmetric

This is the principle underneath both stories. A deadline that is too short and a deadline that is too long fail in completely different ways, and treating them as two sides of the same dial gives you the wrong number every time.

Too long: an operation that has genuinely died holds a row open for a while longer. Somebody waits. Nothing is corrupted, nothing is duplicated, and the eventual cleanup is uneventful.

Too short: you record a failure for work that succeeded. Now your records and reality disagree, and every recovery path you have (retry, recreate, refund, alert) is aimed at a problem that does not exist and will make a second one.

Those are not equally bad, so they should not get equally sized margins. For anything that creates, moves or deletes real resources, we take the generous side deliberately and write down why.

Two clocks that start at different moments

There is a subtlety in any queued command system that is easy to miss and easy to measure. Our platform computes the deadline as an absolute moment, at the instant the command row is written. The agent computes its deadline as a duration, starting when it begins executing.

Those are different clocks. If the agent is offline, or busy, or the command sits in the queue for thirty seconds, the platform has already spent thirty seconds of its budget and the agent has not. The platform gives up first.

That is the right way round, and it is worth being explicit about why. The platform's deadline answers the question the customer is actually asking: how long since I clicked? The agent's answers a different one: how long may this piece of work run? Wall clock from the click is the number a support conversation is about. Making them the same number would be neater and would answer the wrong question.

A third clock hides in there too. On our agent, waiting for a concurrency permit happens outside the execution deadline. A command queued behind another command against the same integration is not burning its own budget while it waits. If your deadlines are tight, check which of your clocks includes queueing; the answer is rarely the one in the documentation.

What the deadline actually cancels

When the agent's deadline expires it signals cancellation and stops waiting. What that cancellation reaches is your own code: the polling loop that was asking the hypervisor how the task was going.

It does not reach the hypervisor. The backup keeps running. It will finish, or it will fail, entirely without us. Nothing about the cancel travelled down to the storage array.

This is not a limitation to be engineered away. It is the honest shape of the problem, and the useful move is to record the result truthfully instead of pretending. The taxonomy that survived contact with production has three outcomes and not two.

• Failed. We know nothing happened. The customer can retry, and there is nothing to clean up.
• Unknown. We stopped listening before there was an answer. Something may exist.
• Needs reconciliation. We have positive reason to think something exists that we cannot account for.

A restore that times out is the sharpest case. There may now be a machine running at the hypervisor that our database has no row for. Marking that failed would be a lie that hides a machine; marking it succeeded would be a lie that invents one. It gets marked as needing a look, and the panel asks the customer to check rather than offering them a retry button that would build a second machine next to the first.

We put that rule in the code as a list of the certain cases, not the uncertain ones. Four specific error codes mean the work never started, because all four come from checks the agent runs before it makes the first call. Everything else, including any error code we add later and forget to classify, defaults to uncertain. The direction matters: a forgotten code that quietly books as failed makes an existing machine disappear from the records, and a forgotten code that books as uncertain only costs somebody a look.

An uncertain result needs a way to become certain

A status that means "somebody should check" is only honest if somebody, or something, actually checks. Ours is a worker that periodically asks each machine with backup history to list its restore points again, so an archive that quietly vanished at the provider becomes a notification rather than a list that got shorter on the day a customer happened to look.

There is a deliberate cheapness constraint on that loop, because each listing is a command running on somebody else's host. It only visits machines that have a policy or at least one known restore point, and it takes a bounded batch per round. When that stops being enough the right fix is a cursor, not a bigger number.

The loop also needs to survive two of your own servers deciding to run it at the same second. We do that with a durable idempotency key made of the machine and the window number, not a timestamp: two servers never tick on the same millisecond, but they are certainly inside the same six hour window, so a timestamp would deduplicate nothing.

What we would tell you to write down

If you are choosing deadlines for remote work, four questions get you most of the way.

1. 01What is the hard ceiling, and who enforces it? If two components disagree, the smaller one wins in practice and the larger one is fiction.
2. 02When does each clock start? Queue time is either in the budget or it is not, and both answers are defensible. Not knowing is not.
3. 03What does expiry cancel? If the answer is "our own waiting", say so out loud, because the rest of the design follows from it.
4. 04Which side does an unclassified outcome fall on? Make uncertainty the default, and let certainty be the thing you have to write down.

None of this makes a long backup shorter. It makes the record of it true, which is the part you will be reading at two in the morning.