Updated at three in the morning, on purpose

The agent that runs in a customer's own network gets a new build every few days. Until this round, updating it was a button somebody had to press, so in practice it happened when somebody happened to look. The wish was one sentence: everybody can set an update schedule on their agent, several times a day, on by default at three in the morning.

The sentence is easy. What made this a round is that a scheduled restart of infrastructure has many ways to be quietly wrong, and almost all of them look fine from the outside.

The default is a column default, and that is the whole wish

"On by default for everybody" has two readings. In application code it applies to agents created from now on, which excludes exactly the customers who already have one. As a column default in the migration it applies to every existing row in one statement.

So it is a column default: a boolean that starts true and a JSON array that starts as a single entry of 03:00. Measured after the migration ran on the development database: every existing agent on, every one at three, and a freshly created agent that passed neither value got the same. That last check is the one that proves the default is doing the work rather than a seed script.

The times are wall clock in the platform's time zone, not UTC. There is no per organisation time zone and the panel renders everything in that one zone, so a schedule stored in UTC would drift by an hour in summer and a customer who typed three would be interrupted at five.

Two layers on one regular expression

A time is HH:MM on a five minute grid, and that pattern exists twice: once in the schema validator that rejects a form field with a readable message, and once as a database constraint that rejects everything which never went through the form.

A check constraint may not contain a subquery, so "every element of this array is a valid time" cannot be an existence test over the unpacked array. What it can be is a JSON path predicate that selects the elements which are wrong, wrapped in a test that the selection is empty.

The first version of that predicate was subtly useless, and it looked complete:

$[*] ? ( !(@ like_regex "^([01][0-9]|2[0-3]):(00|05|10|15|20|25|30|35|40|45|50|55)$") )

A regular expression match against something that is not a string is not false in JSON path. It is Unknown. And the negation of Unknown is also Unknown, and a filter keeps only elements for which the predicate is genuinely true. So the selection of "wrong elements" came back empty for an array of numbers, and the constraint accepted it. Measured against PostgreSQL 17.10, all three of [3], [null] and [{"a":1}] went straight through.

The fix is one clause in front: elements whose type is not string are wrong by definition. All three are refused now. This is the exact failure class that keeps costing us rounds: a guard that catches nothing while looking complete, and no test fails, because there was nothing to fail against.

Five steps, and the order is the round

Every minute, a worker asks who has a schedule, hands the rows to a pure planner that decides which slot is due, and then walks five steps per agent. The claim is step four and not step one, and every step in front of it is there for a reason.

The first two rows are deliberate opposites, and that asymmetry is the most interesting decision in the round.

Nothing to do consumes the slot, because the answer cannot change within one slot: the release manifest on disk is the same for the whole window, so re asking every minute costs queries and buys nothing. And it gives the customer something to look at. An agent page that says "03:00, already current" is distinguishable from a schedule that is not running. An empty list is not.

Offline consumes nothing, because a slot claimed while the host happens to be rebooting is consumed forever, and then that customer silently missed a night. The next minute tries again, until the catch up window closes; after that the agent simply takes its next slot.

Fifteen minutes, and not six hours

Backups in this platform have a catch up window of six hours. This one is fifteen minutes, and the difference is what goes wrong when you are generous.

A backup that is taken at nine in the morning instead of three is still a backup. An agent that restarts at nine in the morning instead of three is an interruption in the middle of a working day, and three in the morning is precisely what the customer chose in order to avoid that. Fifteen minutes covers what the window is actually for, a gateway that just restarted or a couple of skipped rounds, and nothing else.

The planner looks two calendar days back rather than one, which is not an off by one. A time of 23:55 with a catch up window falls, just after midnight, into yesterday's calendar day; with only today in scope that slot could never come due.

Two rate limits, for two different reasons

Five agents per organisation per round, and twenty five in total.

The per organisation limit is about the customer's bandwidth, not ours. The binary is around twenty two megabytes, a hoster with forty agents typically has all forty on 03:00 because that is the default, and forty simultaneous downloads leave through one outbound connection. Five a minute spreads a fleet of forty over eight minutes, comfortably inside the fifteen minute window.

The global limit exists because the first one is per organisation: a hundred organisations would otherwise mean five hundred updates in a single minute. The remainder goes in the next minute, still inside the window.

There is no rollout flag on any of this, which is unusual for us. The switch is already at the customer, on the column; a second platform wide flag would promise something the column already promises. What does exist is an emergency brake for us: setting the poll interval to zero stops the timer without changing a single customer setting.

The night with two threes

On the last Sunday of October the wall clock time 03:00 exists twice, and on the last Sunday of March 02:30 does not exist at all. Both are answered by one shared helper, not copied into this module, precisely because a daylight saving algorithm maintained in two places will disagree on one day a year.

The helper resolves an ambiguous wall clock time to the first of the two instants, and a non existent one to the next instant that does exist. The consequences are the ones you want. On the long night, 03:00 runs once and not twice; measured, one slot. On the short night, 02:30 becomes 03:30 local rather than being skipped, because not skipping is the safe side of that particular coin. Both are pinned by tests.

Days are walked as calendar days rather than by adding a fixed number of milliseconds: adding twenty four hours across a transition either skips a day or does one twice.

Who gets told, and how often

The reconcile that reports outcomes lives in the worker rather than on the dispatch path, and only the schedule notifies. A manual click already had a person watching it.

Notifications group per organisation, because forty agents on release day should not be forty rows in an inbox. A failure is reported at most once per agent per calendar day. And the timestamp recording "we considered notifying" is written even when the decision was to stay quiet, so silence is distinguishable from a worker that never got there.

A scheduled update deliberately does not appear as a root operation in the task drawer: at three in the morning nobody is waiting, and a drawer full of overnight rows is useless for the thing it exists for. The trail is a run row, an event and a notification. That event is its own kind, because the timeline must not claim the customer personally started something at three.

What we would write down

1. 01"By default for everybody" is a column default. In application code it means "for everybody from now on", which excludes the people you were thinking of.
2. 02Unknown is not false. In JSON path, a predicate on a value of the wrong type is neither true nor false, and negating it does not help. Test the type first.
3. 03Decide per situation whether it consumes the slot. Nothing to do and cannot reach are opposite answers, and getting either backwards is invisible.
4. 04A catch up window is a promise about interruption, not about completeness. Copy the number from a system with a different failure mode and you get the wrong one.
5. 05Rate limit for whoever actually feels it. Forty agents in one customer network share one uplink, and that is the constraint, not our download route.
6. 06Solve daylight saving once. Two copies of the same reasoning agree until the last Sunday in October.

For clarity about what this is: the schedule is live and on by default. Several things are unproven. The worker was not exercised inside a genuinely running gateway, no real agent has performed a scheduled update yet, the failure notification and its daily de duplication were not observed end to end, and the screen was verified in two languages of nine.

From the platform work on Provibr. Why an agent refuses an update it cannot verify is in how a signed agent update refuses itself, and why the agent connects outward instead of listening is in outbound agent instead of port forwarding.