Skip to content

Blog ·

A ledger that counts before anyone is billed

Monitoring produces a number for every check long before there is any price. Here is how we built the counting first, and why that order is the safe one.

By Provibr

There is no price on monitoring. No amount, no currency, no plan, no invoice line. What there is, since this round, is a ledger that records every check the platform actually sent, keeps it thirteen months, and closes each hour into a bucket that is never edited again.

Building the counting before the charging looks like the wrong order until you have done it the other way once. A meter written alongside an invoice inherits every convenience the invoice wants: a retry quietly folded in because the total looked odd, a correction applied by editing the row. Those are decisions, and they are easier to make well when no total depends on them.

So the ledger exists, it is empty of money, and one column in it is deliberately blank.

What counts is what actually left

A scheduled check moves through a small set of states, and the ledger asks one question per state: did traffic leave the building. That is a table over every outcome rather than a switch with a default branch, because a default branch is how a seventh outcome silently becomes "does not count" and a month goes missing.

OutcomeRow writtenCountedWhy
schedulednonothe slot is reserved, nothing has been sent
claimednonoclaimed but not dispatched; the row appears at dispatch
dispatchedyesyesa check genuinely went out
completedyesyesfinished; it already got its row when it was dispatched
missednononever sent: no agent, no healthy probe, maintenance, or a slot that passed
expirednonothe deadline passed without anything leaving

Two of these rows deserve their own sentence.

A check that reaches a dead machine counts. There was traffic and there was a measurement; that the answer is "down" is the result, not the question. A rule that only counted successful checks would charge least for the customers we work hardest for.

And the expired outcome is subtler than it looks, because a request can expire after it was sent. It does not double count, and not by luck: a tick only reaches it through the sweep that touches ticks which never made it past dispatch, and anything that did leave counted as dispatched and already has its row.

The source class, our hosted probe or the customer's own agent, is carried in the row and in the signature of the function that decides. Both count the same today, and precisely because they do, the class is written down rather than assumed.

One tick, one row

Failover is the case that quietly inflates a usage number. A check runs from one source, that source is unavailable, another takes over, and now there are two attempts for one measurement.

Here it is a unique index on a key of the form tick:<tick id>. Retries and failovers within the same tick produce the same key, so the second insert conflicts and returns nothing. Both write paths use that; neither looks first, because between a lookup and an insert there is room for a second gateway.

The key is a text column and not the tick's foreign key, and that is the interesting part. Ticks are kept seven days; usage thirteen months. When the sweep removes the tick, the reference is set to null, and in Postgres a null is never equal to another null, so a unique index over that column would quietly start allowing duplicates a week after the fact.

A manual test is traffic too

There is a button that runs a check right now. It does not count, and it is still recorded.

Both halves matter. It does not count because a person pressing a button is not the scheduled service. It is recorded because it is still traffic to somebody's machine, and a thousand presses an hour is exactly what a usage system should be able to show you. A second table nothing reports on would hide the only case where the number is interesting.

What keeps the two apart is not a convention but a database check: a row of the manual kind must not count and must carry no tick. "Manual and counting" cannot exist, which matters because such a row would look completely ordinary in every report.

An hour closes late, and then never changes

Buckets are per organisation, per hour, per source class, per check type. An hour is finalised only once fifteen minutes of the next hour have passed.

The obvious reason, a late result, is not the real one; results do not affect the total. What can genuinely be late is the row itself: a transaction committing at 10:59:59 becomes visible at 11:00:00, several gateways each read their own clock, and one that was briefly away catches up afterwards. Fifteen minutes is an order of magnitude more than anything a single check may take, where the maximum timeout is 120 seconds and the lease is 45.

Once an hour is closed, it is not edited. A late event produces a new row pointing at the base row with a machine readable reason, never an update. That is enforced by shape rather than by discipline: the unique index over organisation, hour, source class and check type is partial, restricted to rows that are not corrections, so a second correction on the same hour does not collide with the first and vanish. If corrections shared that index, the exact evidence you would want during a dispute is what would be lost. The reason itself is one token and not a free sentence, because the panel translates it into nine languages and an export groups on it.

Limits that are not a price list

Every organisation has an allowance, and every number in it was chosen to stop an accident rather than to describe a package.

At most 200 rules, deliberately the same number the create action already enforced: a lower default would have pushed existing accounts over their limit on day one and fired the pausing logic below with no announcement. At most 9 sources per rule, which is the eight hosted locations plus one agent the schema already allowed. All known check types, derived from that list rather than typed out, so a seventh type is not forbidden by default for everybody. A floor of 60 seconds, which an allowance may raise and can never lower; lowering it is a separate paid entitlement that has a name and does not exist.

The revision number in that row is what ends up in the ledger, and zero means "there is no row, this is the default". Without that distinction a usage row from last year cannot be traced back to what was permitted at the time, which is the entire point of storing a revision.

What pauses when the allowance shrinks

If someone reduces an allowance below what an account is using, some rules stop. Which ones is a decision, and there were two candidates.

Most expensive first is unavailable, because there is no price. It would mean deriving "expensive" from the number of sources or the interval, and that pauses the sharpest monitoring the customer has. Somebody reducing their plan would rather lose a peripheral check than the one that wakes them up.

So the oldest rules survive and the newest ones pause. The oldest carry the longest incident history and the alerts people rely on, and "the last thing you switched on is the first thing to go" is the only ordering a customer can predict without reading our code. The sort key is creation time and then rule id, two keys because a template application creates several rules in the same millisecond; without the tiebreaker the preview would not match what happens next.

Rules that break a limit on their shape, a forbidden check type, hosted sources that are no longer allowed, too many sources, are removed first and then counted towards what remains. An interval that is now too fast pauses nothing at all: a raised floor refuses a new or edited rule and leaves running ones alone until somebody touches them.

The empty column

Buckets have an invoice_ref column and it is empty. It is a named seam and not an oversight, in the same spirit as the entitlement key that has a name and no implementation: when billing arrives, the shape it needs is already there, and nothing about the counting has to be revisited under deadline.

Two related decisions follow the same reasoning. The ledger has its own table rather than being derived from the ticks, because ticks live seven days and this lives 395. And the per check rows are left out of the personal data export, with the reason written next to the omission: hundreds of thousands per month, an export capped at 25,000 rows per section, and hourly buckets carrying the same totals. The buckets and the allowance do go in.

What we would write down

  1. Write the meter before the price. Every question is easier to answer honestly when no total depends on the answer.
  2. Record when the work leaves, not when the result arrives, and a late answer stops being a metering problem.
  3. Make the idempotency key outlive its subject. A key built on a row that gets swept away starts permitting duplicates on a schedule.
  4. Enumerate every outcome in a table. A default branch turns a new state into a silent zero.
  5. Corrections are rows, never edits, and the uniqueness protecting the original must not also block the correction.
  6. Choose the ordering your customer can predict, and write down the one you rejected.

For clarity about what this is: monitoring sits behind a rollout flag that is off by default, there is no entitlement layer with money in it, and none of this is purchasable today. Two things are unproven. The retention sweep has never run against a genuinely old row, only against the schema. And the worker that closes hours was exercised through its function rather than a long running process, so a gateway that is away for a night and catches up is reasoned about and not observed.

From the monitoring work on Provibr. Why a probe holds no privileges it does not need is in a probe that cannot be talked into anything, and why an incident does not close just because the measurements stopped is in an incident that never closes on silence.

Start with one host and one agent

Create an account, install the agent with a single command and connect your first Proxmox host. The trial runs for 14 days and asks for no payment details.