A scan that closed its own tick

Our exposure scanning has a scheduler. It claims a slot, dispatches a scan command to the agent that can reach the machine, and waits for two things: the measurement itself, and the completion of the command that carried it. When a slot ends without an answer it is marked expired, and the machine shows up as "no recent measurement".

Every successful scan was marking its own slot as expired. Four out of four, measured against a real agent, while the four measurements themselves were read in and stored perfectly. The panel reported that four machines had not been scanned recently, minutes after scanning them.

One word, from a different table

The handler that closes an unanswered tick began with an early return: if the command succeeded, there is nothing to close, so leave.

The word it compared against was succeeded. That word does not exist in the command status enum, which reads pending, sent, completed, failed, timeout, cancelled. It is the word from a different table entirely, the one that tracks agent installations, and it arrived here because somebody wrote the handler while looking at that other table's states and because the parameter was typed as a plain string.

A plain string is the whole story. The comparison compiles against any text at all, so nothing objects, and the failure is not an exception but a branch that is never taken. The handler ran its "no answer arrived" path against every successful scan and dutifully marked each slot expired.

No unit test could have found this. There is no command lifecycle in a mock: the mock is the thing that hands you the status, so the status is whatever the test author wrote, and the test author wrote the same wrong word. Only a real agent, sending real completions through the real dispatcher, produces the state that exposes it. The parameter is now typed to the three statuses that can actually reach this function, which is how the equivalent monitoring handler had always been written. Two functions, the same question, and only one of them had been written carefully.

And the two messages arrive out of order

Fixing the comparison was not enough, and the reason is a second thing that only a real agent shows you.

The agent sends two separate messages: the scan result, and the completion of the command. They travel independently. Measured, the result arrived first, fourteen milliseconds before the completion. That interval is small enough that reasoning about it never happens and large enough that it always occurs.

So by the time the completion handler runs, the tick has usually already been closed by the result. An update without a condition would reopen a finished tick and mark it expired all over again, which is the original bug wearing a different hat. The update is therefore clamped to the two states in which a tick is still waiting for an answer. If it is already resolved, the handler changes nothing.

That is a general shape: when a fact can arrive from two directions, the writer that is not the primary one has to state which states it is allowed to move out of.

One closed finding that stopped the whole fleet

The second defect is quieter and it did more damage.

When a vulnerability feed updates, open findings are reassessed against the new data. A finding whose match has disappeared is closed, and it is closed as unverified rather than resolved, because nothing was measured: what changed is our inference, not the machine. That distinction is the reason the status exists.

The reassessment fetched the existing derived findings so it could tell "update this one" from "create a new one". It fetched only the open ones. That is a reasonable looking filter and it is wrong, because the uniqueness constraint underneath knows nothing about status: it is on the server and a fingerprint of the finding, full stop.

So the sequence goes: a national vulnerability database corrects a version range, our match disappears, the finding is closed. Weeks later the same database revises the range back, which it genuinely does, and the match returns. The query does not see the closed row. The code takes the "this does not exist yet" branch. The insert hits the unique index and raises a constraint violation.

And then the violation escapes. The caller catches it as "the advisory round failed", and that round reassesses nothing at all, for every customer on the platform. The condition that caused it does not go away on its own, so every subsequent round failed the same way. One finding that had closed and reopened froze the reassessment of an entire fleet, permanently, with a log line as the only symptom.

The fix is small: include the closed rows, and when a closed finding matches again, reopen it with its timeline intact rather than starting a new one. A reopened finding is a different fact from a new one, and the person triaging it deserves to see that this came back rather than arrived.

A feed loop, measured three times

The feed worker was written before there was any way to run it without touching a real database, which is the trap the previous security round fell into: importing the database package reads the environment and connects, so a unit test with an injected fetch quietly wrote four advisories into a development database while reporting green.

The storage side is now a seam, and the loop is testable against an in memory ledger while production code uses the real one unchanged. Then it was run three times against the actual public feeds.

The 304 is the point of the exercise. A statically served advisory catalogue answers a conditional request with "nothing changed", and the correct behaviour is to keep the stored validators, write nothing, and count the round as a success.

Some sizes worth knowing, since they are the ones people guess at: the known exploited catalogue is about 188 kB on the wire because it is compressed, and one page of two thousand records from the larger feed is roughly 18.7 MB. Both are read through a hard byte ceiling rather than through the ordinary JSON helper, which has no limit at all; these are the two largest responses the platform ingests and they come from hosts we do not run.

A window that halves instead of skipping

The larger feed is paged, inside a moving time window. Two things were wrong with that loop, and both of them lost data without ever raising anything.

If a window contained more records than the page ceiling allowed, the loop stopped and moved the cursor to the end of the window anyway. Everything past the ceiling was skipped, silently, and would only ever be picked up if some other change touched the same records again.

And an empty page in the middle of a window, with more records still claimed to exist, advanced the cursor by a full page. Two thousand records gone.

Both are now handled by the same principle: when the answer does not fit, ask a smaller question, and when the answer is impossible, do not move.

An oversized window is halved rather than truncated, up to six times, down to a floor of one hour. Six halvings turn a week into a bit over two hours, which is enough for any realistic backlog. Below an hour it stops: a window of minutes that still returns twenty thousand records is not a window problem, it is an outage, and refusing to write while leaving the cursor where it is happens to be the only honest answer available. An empty page mid window is treated the same way, as an error that leaves the cursor alone, so the next round tries again rather than continuing past a gap.

The halving happens after the page is written, not before. What we already read belongs to the original window and is valid; discarding it would trade one kind of data loss for another.

One small thing in the same area, worth repeating because it is easy to get wrong under time pressure: the API key goes in a header and never in the query string. URLs end up in logs, in proxy caches and in error reports. Headers do not.

What we would write down

1. 01Type the parameter, especially when it is a status. A string comparison against a word from another table's enum compiles, runs, and takes the wrong branch forever.
2. 02Two messages about one event will arrive out of order. Clamp the secondary writer to the states it may move out of.
3. 03A uniqueness constraint that ignores status will meet code that does not. Fetch the closed rows too, or the first reopening becomes a violation.
4. 04Ask what one failure poisons. An exception escaping a per item loop can turn a single bad row into a platform wide stop, and the condition does not clear itself.
5. 05A conditional request answered with 304 is a success. Keep the validators, write nothing, and do not count it as a failure.
6. 06When a window is too large, halve it; when a page is impossible, stop. Advancing a cursor past data you did not read is the one outcome with no way back.
7. 07A worker that cannot be run without a database will be tested against your database. A storage seam is cheaper than the cleanup.

For clarity about what this is: exposure scanning is built and sits behind a rollout switch, so no customer has it in front of them yet. The reassessment, the feeds and the fan out to multiple addresses are exercised; what remains unproven is the window narrowing against a live feed rather than an injected one, index behaviour on a genuinely full catalogue, and a scan running from a machine other than the one that requested it.

From the platform work on Provibr. Why the matcher refuses to be sure about a version is in a match that refuses to be sure, and why the probe behind it sends almost nothing is in why a vulnerability scanner should send zero bytes.