Blog ·
Two rounds, one field number
Two branches added a variant to the same protobuf message on the same day and both picked the same number. Very little about that fails loudly.
By Provibr
Two pieces of work ran in parallel on the same repository. One added the ability to ask a hypervisor what became of a background task. The other added the ability to reinstall a container. Both needed a new variant in the same protobuf message, both looked at the file, both saw that the highest number in use was thirty three, and both wrote thirty four.
Neither was careless. Each read the file correctly at the moment it branched. The collision exists only in a third state neither of them ever compiled: the merge.
| Where the collision shows up | What happens |
|---|---|
| The two branches separately | Nothing. Each is correct against its own starting point |
| The merge, in the schema file | A conflict, and then a hard compiler error if both are kept |
| The merge, in the committed generated code | A conflict, and then nothing at all if both are kept |
| The compatibility checker | Green in both branches, because it compares each one to its own baseline |
| A deployed agent and gateway that disagree | Never reached here: a command that would arrive empty, reported layers away from the cause |
What the wire actually carries
A protobuf field name is a convenience for humans. What goes over the connection is the number, shifted left three bits with a type indicator in the low bits, so field thirty four with a length prefixed payload is written as the byte sequence for two hundred and seventy four. Rename a field and nothing changes on the binary wire. Change its number and every deployed reader is now decoding something else with the same key.
That is why a number, once it has gone out in a release, is frozen. It is not a style rule. It is the only identifier the two ends share.
What the tooling says, and what it does not
Git noticed. The conflict showed up in the schema file and in the committed generated TypeScript, both times as overlapping additions in the same place. The tempting resolution is the one that looks obviously correct for two independent features: keep both.
In the schema file that is caught. Compiling it produces a flat refusal naming the message and the field that already holds the number, which was measured rather than assumed: a two variant test message with both variants on thirty four is rejected with a line and column and a suggestion of the next free number.
The generated TypeScript is a different story, and it is the reason this is worth writing down. That file is committed, because everyday work should not require the schema toolchain to be installed. The cost of that choice is that it takes part in merges like any other source file, and it is ordinary code. A duplicate branch in a switch statement is legal TypeScript. Compiling a small file with two identical case labels, both carrying the tag check for field thirty four, produces no diagnostics at all and exits cleanly. The second branch is simply never reached.
So a hand merged generated file gives you an encoder that writes the same tag for two different messages and a decoder that always resolves it to whichever one is listed first. Nothing fails at build time, nothing fails at startup, and nothing fails on the first message. One of the two features is simply wrong, in a way that only shows up on the machine at the other end of the connection.
Why the compatibility check was green
The schema is guarded by a breaking change checker, and it did its job. It compares a proposed schema against a baseline. Adding a new variant to an existing choice is not a breaking change, and it was not one in either branch, because each branch's baseline genuinely did not have thirty four in use.
This is the general shape of the problem and it is worth stating separately from protobuf: a check that compares a branch to its own starting point cannot see a conflict that only exists between two branches. The same class of bug turns up wherever parallel work picks a value out of a shared, ordered space. Database migrations here are chosen by the timestamp in their journal, and those timestamps are round numbers written by hand rather than a real clock, so two rounds each writing one pick their own; a migration whose timestamp lands below one that has already been applied is skipped permanently, and the migrator still reports that the schema is up to date. Different mechanism, same silence, same cause.
Why silence is the expensive part
If both numbers had survived into a release, the failure would have been remote and mystifying. Protobuf discards a choice field it does not recognise, so an agent built from one branch receiving a command from the other does not get an error. It gets a command with nothing in it, and reports something like "the message carried no instruction". The cause is layers away from the sentence, on a machine belonging to somebody else, and the person reading it has no reason to suspect a number.
The platform already refuses to send instructions to agents that have not announced the matching capability, precisely because this failure mode is so unhelpful: an agent that is merely too old at least produces a refusal the platform wrote, naming the feature and the version. That guard covers version skew. It does not cover two numbers that mean two things, because from the platform's side nothing is out of date at all.
The rule that came out of it
A field number that has gone out in a release is frozen, so the round that has not shipped yet is the one that moves. Thirty four had already left in an agent release as the task status variant, so at the merge the container reinstall variant became thirty five, and the generated code was regenerated from the schema rather than merged by hand. Regeneration is the important half: a hand merge of generated output is a hand written encoder that nobody reviewed as one.
The practical instruction for whoever adds the next one is short. Read the highest number in the main line at the moment you commit, not at the moment you branched. And every merge that touches the schema gets that checked deliberately, by a person, because no tool in the chain is looking at it from that angle.
Reservations, and why they are worth honouring
The same schema shows what the discipline looks like when it works. Three numbers sit reserved for integrations that were planned and never built, so nobody can quietly reuse them. Two more were held for two specific control panels, and both were eventually filled by the rounds they were promised to. A third panel that happened to be finished earlier did not take one of them; it took the next free number further along instead.
That is the whole point of a reservation. Eating one because you happened to be ready first is exactly how a numbering scheme stops meaning anything, and the comment in the file says so in as many words.
Filling a reserved number as intended does get reported by the breaking checker, as the removal of a reservation. That report is expected and correct to accept: no field ever carried that number, so no deployed reader can have an opinion about it. Worth knowing in advance, because it looks alarming the first time and the temptation is to pick a different number to make the tool quiet, which throws away the reservation you were honouring.
What we would write down
- Treat a shipped number as immutable, and treat "the highest number I can see" as a statement about your branch rather than about the project.
- Read the shared space at commit time, not at branch time. That is the only moment the value is actually being claimed.
- Regenerate generated code, never merge it. A conflict in generated output is a signal to run the generator, not an invitation to edit.
- Know which of your checks are baseline relative. They cannot see collisions between siblings, and that gap needs a human at the merge.
- Reserve the numbers you have promised or planned for, with a comment saying who they belong to, and accept the noise that filling one correctly produces.
- Look for the same shape elsewhere. Any hand assigned value in a shared ordered space, migrations included, has this failure and usually has it silently.
For clarity: nothing was shipped with the collision in it. It was caught at the merge, the later variant moved, the generated code was rebuilt from the schema, and both features went out as separate agent releases hours apart. What made it catchable was that somebody read the conflict instead of resolving it, which is not a process you can automate but is one you can write down.
From the platform work on Provibr. The database version of the same failure is in the migration that never ran, and why a signed agent update refuses itself rather than trusting the sender is in how a signed agent update refuses itself.
