An incident that never closes on silence

Every monitoring system eventually faces the same question in a slightly different costume: what do you do when the measurements stop arriving?

The tempting answer is to treat it as recovery. Nothing is failing any more, the red row goes away, the dashboard is calm. It is also the single most expensive lie the system can tell, because the moment you stop being able to look at something is exactly the moment you must not claim it is fine.

This is the design we landed on for incidents, and most of it is about keeping "we measured that it is broken", "we measured that it is fixed" and "we cannot see it right now" as three separate statements that never quietly become each other.

Why the incident is an object

Before this, an outage was a series of rows in a state transition table. That is enough to draw a graph and not enough to say anything. Three things only exist once there is an object.

A duration. The recovery message used to carry no duration at all, because there was nothing to fill the parameter with. An incident has a beginning, so the duration is a measurement rather than an estimate.

A human layer. Acknowledging, assigning and closing by hand are things people do, and they must not touch the measured state. An acknowledged_at column on a measurement would blur exactly the line this whole feature is built on.

Deduplication over time. One notification when it opens, reminders on their own clock, at most one escalation. Without an object there is nothing to hang "already told you" on, and then every tick is another chance to tell you again.

The deduplication is an index

The uniqueness that stops two incidents existing for one rule is a partial unique index on the rule, where the resolution timestamp is null. Two gateways processing the same transition produce one incident by construction; the second gets a unique violation and moves on.

Code that checks whether one already exists and then inserts is two statements, which is two chances for something to slip in between. This is the same shape as the scheduler's slot index and the rack occupancy index elsewhere in this product, and for the same reason.

There was also a genuinely embarrassing bug here, worth writing down because it is invisible in review. The deduplication key for the notification was built from the rule, the new state and the current timestamp. A key containing the current timestamp deduplicates nothing, by construction, and it looks entirely reasonable on the screen. It is now the incident identifier. Measured afterwards: three consecutive failures produce one incident and one notification, and three more change nothing.

Three ways an incident can end, and only one is a measurement

The middle one is the point of the list. A rule that gets replaced by a template revision has to close its open incidents, and it must do that without a false recovery measurement. If the only available resolution were recovered, replacing a template would silently write "this came back up" into the history of every service that was down at the time.

There is deliberately no stale resolution. That measurements stopped arriving says nothing about the target.

Which transitions touch an incident at all

An incident hangs off a statement about the target. So a transition into "source unavailable", meaning our own agent went away, opens nothing, and, more importantly, closes nothing. If a departing agent could close an incident, a live outage would vanish from view at the exact moment we lost the ability to look at it.

A transition into "configuration error" behaves the same way: the rule no longer describes anything measurable, which is a statement about us, not about the service.

"Degraded" does close an incident. The service is answering, the condition counts as healthy, only the latency is high. Leaving it open would make a slow service a permanently open outage, reminders included.

Under this, from an earlier round, sits a hard distinction between an unreachable source and a target that is down. A check whose agent is not connected produces "source unavailable" and never "down". Everything above is that same rule, one layer up.

Maintenance silences notifications and keeps measuring

A maintenance window suppresses notifications. It does not suppress measurement. The incident still opens, the timeline still runs, and only the message stays in the drawer.

That is not a detail. The question people ask after a maintenance window is "was it back on time", and that question is unanswerable if you stopped measuring while it mattered. There is a separate, explicit checkbox that does pause the checks, and it is off by default; with it, the scheduler skips claiming the slot, so nothing becomes due and nothing is missed. What is left afterwards is a single gap marker, the same as for any other gap. There is no catch up.

The window itself is stored as an absolute interval with an IANA time zone beside it. The customer enters wall clock time and wants to read it back as the same wall clock time, even though the panel renders in a fixed zone. Storing wall time plus zone instead would force every reader, scheduler, ingest and screen alike, to redo the conversion, and then a daylight saving transition is an hour of disagreement between two of them.

Where two windows overlap, the one that pauses checks wins. When a customer has made a stricter choice explicitly somewhere, that is the choice to honour.

Reminders, escalation and the order of two statements

Both the reminder interval and the escalation delay live on the rule revision, not on the incident, because they are configuration and configuration belongs with the revision that recorded it. Change the policy during an outage and the running incident stays under the policy it opened under.

Both are off by default. A default that wakes people at night is a decision the customer did not make.

Acknowledging silences reminders, and that is the entire function of acknowledging. Somebody saying "I am looking at it" should not be interrupted every few minutes. The recovery notification still arrives, because that is the answer to the question they were looking at. The reminder clock also runs from the last reminder rather than from the start, so a worker catching up after an hour does not deliver twelve reminders at once.

The worker writes the timestamp first and sends the message second, and the update requires the state to still be what it read. Lose that race and nothing happens here. In the other order, a lost race produces a message with no record of it, and then the message arrives again next round.

Escalation happens at most once per incident, and the check for that lives in the shared rule rather than in the worker. Without it, every pass would write another escalation event for as long as the outage lasted: thirty an hour on a two minute poll.

Flapping is the exception, and it is decided when the incident opens: three incidents on the same rule within an hour. A flapping check by definition never has an incident that stays open for long, so any time based escalation misses it forever. Three and not two, because twice is bad luck and a threshold of two would call every ordinary service restart a flap. An hour and not a day, because three outages spread over a working day are three outages.

The duration is a number and a unit

The recovery message carries how long it lasted, as a value and a unit, never as a sentence. If the worker composed "1 hour 12 minutes" itself, that English fragment would sit in the middle of a translated sentence in eight other languages. What travels instead is a unit that the translation selects on, with a plural inside each branch.

One unit, no compounds: a compound needs a separator and an ordering, and those differ per language. Seconds below a minute, minutes up to two hours, hours up to two days, days after that. Ninety minutes reads more sharply than two hours, and rounding upwards would make an 89 minute outage last two.

Two honest consequences we did not fix

Down, recovered and reminder share a notification group key, because a reminder is literally the same subject again and should update the existing row rather than adding one every time the clock fires. The cost is that a reminder updates a row somebody has already read, and is therefore invisible to them. Escalation gets its own group for exactly this reason: the message that has to land hardest must not be the one overwritten first. Monitoring notifications do not go out by mail today.

What we would write down

1. 01Never let absence of data close anything. Give "we cannot see it" its own state and make sure it neither opens nor closes an incident.
2. 02Give the incident an object as soon as you want a duration, an acknowledgement or a second message about the same thing.
3. 03Put the deduplication in a unique index, and check that your notification key does not contain the current time.
4. 04Enumerate how an incident can end, and require evidence only where evidence is claimed.
5. 05Suppress notifications during maintenance, not measurements, and make pausing the checks a separate, explicit, default-off choice.
6. 06Update the state, then send the message. The other order loses messages silently.

This work sits behind a rollout flag that is off by default, so none of it is something you can buy today. It is written up as method rather than as an announcement. The measurement layer underneath it, including why an unreachable agent is not a failing service, is a separate story, and the closely related question of when a platform is allowed to state a time at all is in a progress bar that refuses to guess.

From the platform work on Provibr.