Why a Proxmox 200 is not a result

You POST a clone request to a Proxmox node. You get back HTTP 200 and a string that looks like line noise. At that instant, exactly one thing is true: the request was accepted. No disk has been copied, no machine exists, and nothing has failed yet either.

Almost every mutating call in the Proxmox API works this way, and it is a perfectly reasonable design for operations that take minutes. It only becomes a problem when the calling code treats the 200 as an outcome, which is the default behaviour of every HTTP client ever written.

This is how we read those tasks, and which parts turned out to be sharper than they look.

The string is an identifier, and it is structured

The thing that comes back is a UPID: a colon separated identifier that names one running task. It carries, among other fields, the node the task is running on, the type of task, the thing it operates on, and the user who started it.

Three of those are load bearing.

The node is the one you have to ask for status. A task belongs to the node that runs it, and a cluster wide question is not the same question. Take the node from the identifier rather than from whatever variable you happened to have in scope, because those two can differ and the case where they differ is precisely the one you did not test.

The type tells you what actually got started, which is not always what you asked for by name. A backup started through the node endpoint comes back with a task type of vzdump, not something guest shaped.

The target is the guest identifier for guest operations, and it can be empty. That is not corruption. A task can legitimately be about a node rather than a guest, so code that parses this field must return "no guest" rather than throwing or defaulting to zero.

Our parser is deliberately narrow about what it rejects: it requires the prefix, it requires at least eight colon separated fields, and it requires the node and the type to be non empty. Everything else is allowed to be missing, because the format has to keep working for task types we have never seen.

Finished is not the same as succeeded

Ask a node for the status of a task and you get two fields that answer two different questions. One says whether the task is still running or has stopped. The other, present only once it has stopped, says OK or carries an error string.

Writing that as two separate predicates instead of one is not pedantry. A single isDone() that conflates them produces the bug where a failed clone is reported as a finished clone, and the machine that does not exist gets a row in your database. We have two functions: one asks whether it stopped, the other asks whether it stopped and the exit status is exactly OK. Anything else is a failure, including exit statuses we have never seen, because a status we do not recognise is not permission to assume success.

Where the task lives is part of the API, not a detail

Backups taught us this one. It is natural to expect a guest operation to hang off the guest: something under the virtual machine's own path. There is no such endpoint. A backup is started on the node, with the guest identifier as a parameter.

That is not a quirk to route around. It changes what you have to know before you can start: which node, which storage on that node, and whether that storage is even allowed to hold backups. A storage that only accepts disk images will accept your request happily and fail inside the task, which surfaces to a customer as an error message from a program they have never heard of. So we filter the target list by what the storage declares it can contain, and drop any storage that is listed but not answering, before offering anyone a choice.

The same rule in reverse applies to placing a disk. Same idea, opposite content type, and worth keeping as two explicit filters rather than one clever shared one.

Polling is a rhythm, not an interval

While the task runs you have to ask repeatedly. The naive choices are both bad: poll fast forever and you are hammering somebody's management API for an hour, poll slowly and every quick operation feels sluggish.

Our compromise is three quick attempts a second apart, then every three seconds. Starting a machine finishes inside the fast window and feels instant. An hour long backup costs a request every three seconds, which is nothing.

The loop ends in exactly three ways, and all three are explicit: the task reports that it stopped, the command's deadline passes, or the whole command is cancelled. There is no fourth branch and no path on which the loop simply continues, which is a property worth arranging on purpose rather than hoping for. What a deadline does and does not stop is its own subject; here it is enough to say the polling loop must have a way to lose.

Most tasks will not tell you how far along they are

This is the part that disappoints everyone building a progress bar. The task status gives you running or stopped. It does not give you a percentage.

Some tasks write progress into their log, as text. A clone emits lines with a transferred count and a percentage in parentheses. So for those, we read the log tail and take the last percentage we find in it. Parsing a percentage out of a log line is not elegant, and it is the only place the number exists.

For everything else we show an estimate, and here the design choice matters more than the formula. Ours approaches ninety percent asymptotically from an expected duration, so the bar always moves and never reaches the end. A bar that sits at 100 percent while nothing happens is worse than no bar, because it tells the person watching that you are done, and then contradicts itself.

The expected duration feeding that curve is a shape parameter, not a deadline. Ours says a backup usually takes about ten minutes and a restore about twenty, on the reasoning that a restore reads the archive and writes the disks and so moves roughly twice the bytes. If a backup runs longer than the estimate, nothing breaks; the curve just stays politely under the line.

The error is in the log, not in the status

When a task fails, the exit status is often close to useless. "unable to create VM" is a category, not a cause. The cause is in the last few lines of the task log.

So a failure report is assembled: the exit status, plus the tail of the log with the progress lines filtered out, capped to a handful of lines and a few hundred characters. Filtering the progress lines matters more than it sounds, because a failing clone can end with twenty lines of transfer percentages and the actual reason above them.

One rule about that text, which we hold to strictly: it is a quotation. It is what the hypervisor said, and it goes into our records and onto the screen unchanged. We do not translate it, we do not tidy it, and we do not paraphrase it into something friendlier. The sentence around it is ours and gets translated for the reader; the quoted part stays exactly as the other side said it, because it is the only string in the whole exchange that a support engineer can search for.

Do not fold your own words into the machine's sentence

The last lesson is the one we did not expect to learn from a task poller.

Our wait loop used to take a free text label for the operation and fold it into three sentences: something in progress, something finished, something failed. It worked, until somebody noticed that several of those labels had been written in our own office language rather than English. Customers were reading half translated sentences in their progress bars, and when an operation failed, that half translated sentence was stored permanently in the error field.

The fix was not to correct the labels. It was to remove the possibility: the operation is now an enumeration, and each case carries identifiers for "running", "finished" and "failed" rather than words. There is no text left to get wrong, and the sentence gets built in the reader's language at the very end. Keeping those catalogues honest turns out to be its own discipline.

The checklist

For any API that answers a mutation with an identifier rather than a result:

1. 01Parse the identifier, do not just store it. It tells you where to ask, and where is not always where you asked.
2. 02Separate finished from succeeded, in the type system if you can.
3. 03Treat an unrecognised terminal state as failure, never as success.
4. 04Give the poll loop a rhythm and a guaranteed way to end.
5. 05Show progress you actually have, and an honest estimate you can never mistake for progress you have.
6. 06Quote the other side verbatim in errors, and build your own sentence around the quotation instead of inside it.

None of these are difficult. All six are things we got wrong first.