A canary that runs against real servers

A monitoring check that logs into a database is a small piece of protocol code with an unpleasant property: almost every way of testing it tests the wrong thing.

The obvious test is a stub server that speaks the protocol. That works, and it proves quite a lot: framing, timeouts, what happens when the connection drops in the middle of a handshake, whether a refusal produces the right error code. What it cannot prove is authentication, and for a structural reason. A stub that recomputes our own scramble uses our own implementation on both sides of the wire. Swap two arguments in a hash and both sides swap them, and the test is green on code that no real server would accept.

That gap was written down as an open item at the end of the previous monitoring round: SCRAM had only ever been measured against an implementation we also wrote. A canary closes it.

What a canary is here

A canary is a small set of tests that run against the real thing rather than a model of it. Disposable containers on our own server, bound to localhost, holding a genuine PostgreSQL 17, Redis 7, MySQL 8.4 and MariaDB 11. The adapters are called directly rather than through the command envelope, because the subject is the wire and not the plumbing around it. Eleven checks, all of them skipped by default and each one refusing to run unless its address is actually set.

That second guard is not redundant with the first. Running the whole suite with the ignore flag lifted would otherwise let a test quietly connect to 127.0.0.1:0 and produce a failure that reads like a bug in our code rather than a missing environment variable.

What the eleven cover:

The cold and warm cache distinction matters because they are two different code paths. MySQL's caching_sha2_password uses a cheap scramble once the server has seen the account before, and asks for full authentication the first time after a restart. Only a real server restarts.

Two defects that no stub could have shown

PostgreSQL, wrong password. When the password is wrong, PostgreSQL does not answer the SASL initial response with a continuation. It answers with an error carrying SQLSTATE 28P01, invalid password. The adapter had a plain destructuring assignment there, which turned anything that was not a continuation into a protocol error: "this service did not answer like a PostgreSQL server". The SQLSTATE was thrown away. A customer whose monitoring credentials had expired would read that their database was broken.

A stub never comes past that branch, because a stub that is testing the happy path sends the continuation. Both places where an error can now arrive, after the first step and after the second, keep it an authentication failure.

Redis, an empty field. The adapter reads exactly two fields out of INFO: the version and the role. It was asking for the server section, and role is not in that section. It is in replication. The field had been empty since the round that introduced it, and nothing said so, because the stub answered with an INFO block we had written ourselves and had helpfully put role where we expected it.

The fix is two separate INFO calls rather than one call naming both sections, because Redis only accepts multiple sections in one INFO from version 7.0 onward. On 6.x that is a syntax error, and since a failed INFO deliberately does not turn the check red, the field would simply have gone empty again on older servers. A silent degradation replacing a silent degradation.

Both bugs share a shape. Neither produced an error. One produced the wrong error and one produced nothing at all, and both were invisible to a test suite that was green.

MySQL, and the crate that was not added

The same round added MySQL and MariaDB as a check type. It has no client library behind it, and that is a deliberate repeat of an earlier decision rather than a new one.

For PostgreSQL, the choice had been to not write our own SCRAM, because SCRAM has a step where the server signs something the client is supposed to verify, and an implementation that skips that step is green against a real server and equally green against a forged one. That is the dangerous kind of shortcut: it never fails loudly.

MySQL has no such step. Native password authentication is a chain of SHA1 hashes and the fast path of caching_sha2_password is a chain of SHA256 hashes, and in neither does the server sign anything we could fail to check. What remains is framing plus two hashes, and both hash crates were already in the tree. A query builder was not needed either: there is one query, SELECT 1, it is a constant, and its expected answer is a constant too.

That last one is a decision that reads like a missing feature. Above a fixed query, a configurable expected value is a knob where every setting other than the default breaks the check, and a knob like that teaches an operator that knobs mean nothing. It belongs there the day a second fixed query exists and not a day earlier. The version is not a second query either: MySQL sends it unprompted in the handshake packet, and the second query is always the one that grows a free text field a year later.

Two things are refused outright. The client capability that lets a server instruct the client to read a file from its disk and upload it is not requested, and the corresponding response byte is treated as a protocol error rather than as a branch that does something. And full caching_sha2 authentication is only performed over TLS: the alternative on a plaintext connection is to fetch the server's public key over that same unauthenticated connection, which anyone in the middle can answer with their own. The refusal names the repair, which is to turn TLS on.

A status page counts ticks, not results

The same round put a public status page in front of all this, and the arithmetic on it needed one decision that is easy to get wrong.

Every scheduled execution of a rule is a tick. A tick can end as up, degraded or down, and it can also end as source_unavailable (our own measuring agent was not there), configuration_error (the rule did not make sense), unsupported (the agent did not know that check type) or missing. Those last four say nothing about the customer's service.

If the denominator of a daily availability figure were every scheduled tick, an outage of ours would lower the number on the customer's own status page. So the denominator is up plus degraded plus down: the ticks where somebody actually looked at the service. If nothing is left of that, the answer for that day is that there is no answer, explicitly not zero and explicitly not one hundred percent. On the page itself those four states render as unknown and never as down.

Degraded counts as half. That is a choice rather than a law of nature: a service that answers all day but sits above its latency threshold is neither down nor fine, and counting it either way makes one of the two meaningless.

Gaps are shown as gaps. A rule created last week does not have ninety days of history, and a bar that fills that in with the neighbouring day's value is asserting something that was never measured.

The part that a green test suite cost us

Every one of the defects above sat under a passing test. The stub suite was not wrong; it was answering a different question than the one anybody thought it was answering. The value of the canary is not that it is a better test. It is that it is a test where the other side of the wire was written by somebody who does not share our assumptions.

That is worth six disposable containers, and it is worth writing down that the containers are disposable: they run on our own machine, they are removed when they exit, and the tests refuse to run at all if the addresses are not there.

From the monitoring work on Provibr. Why a hosted probe accepts exactly one kind of message is in a probe that cannot be talked into anything, and why a monitoring secret only ever travels once is in a secret that travels once.