A wrong secret that looks like silence

Adding a monitoring check for a new protocol looks like a dependency question. There is a crate for MongoDB, a crate for BSON, a crate that speaks the SQL Server wire protocol. Add three lines to a manifest, write forty lines of glue, ship it.

We wrote all three protocols by hand instead, and the reasons are worth writing down because none of them is "not invented here". Two are measurements, and the third is a promise we had already made to customers about what this agent can do.

What a dependency actually brings

The BSON crate pulls in forty one crates. That is measurable and it was measured, but the crate count is not what decided it.

The check reads four fields out of one server reply. A generic document type holds the whole reply in memory: the host list, the tags, the last write timestamp, the cluster time, everything the server felt like sending. At that point "we only read four fields" is an agreement between developers rather than a property of the code, and agreements are what get broken by the next round that needs a fifth field for a good reason.

So there is a reader that walks the elements, hands back the four that were asked for, and skips the rest without putting it anywhere. It cannot hold the reply, so the promise is a shape rather than a habit. Same decision as the allow list on the Redis INFO command, and the same decision that kept a full database driver out of the PostgreSQL check: a driver brings a query builder, and then there exists, inside the binary running on a customer's network, code that can ask an arbitrary question of a customer's database.

The SQL Server crate is the sharper version. It builds for our release target, which was the thing we expected to block it. What it brings is a second TLS stack: an end of life release line alongside the current one we already ship. The process wide cryptography provider is installed for the current line and does nothing for the older one, so TLS under one check would use a different provider than every other connection this agent makes; a vulnerability in that line is one we ship and cannot fix without waiting upstream; and the certificate policy, which lives in exactly one module precisely so that it cannot drift, would get a second implementation against a different API.

Skipping is where a hand written parser goes wrong

If you write your own reader for a binary format, there is one mistake that matters more than the rest, and it is not in the parsing. It is in the skipping.

Every value in BSON has a type byte and a length that follows from it. A reader that knows the types it happens to have seen, and skips an unknown type by zero bytes, does not fail: it carries on reading the rest of the document from the wrong offset. Everything after that point is shifted. You get plausible values that are not the values that were sent.

So the length table is complete, with a closed refusal for anything outside it. An unknown type byte is a protocol error and never a skip.

The same class of thing sits one level up in how numbers are read. A MongoDB server answers ok: 1 as a double and an error code as a 32 bit integer, in the same document. A reader that only understands 32 bit integers silently misses the first one, and then every failed login reads as a success. The reader accepts a BSON number as any of its three widths and hands back one integer.

The authentication step that has to be able to fail

The check logs in with SCRAM-SHA-256, and not with SCRAM-SHA-1, for two reasons that point the same way.

The older mechanism does not feed the password into SCRAM. It feeds a hex encoded MD5 of the user name, a fixed string, and the password. Supporting it would mean writing code that makes an MD5 of credentials, for a mechanism no current server needs to offer.

The quieter reason is worse. A server that offers both would, on a fall back, end up on the weaker one without anything turning red, and then there is a green measurement sitting above a login that is weaker than the customer believes theirs to be. If the server does not offer SHA-256 for this user, the check refuses with its own code and says so.

Writing SCRAM by hand was refused in an earlier round for the PostgreSQL check, and the objection was real: the exchange has a server signature the client is supposed to verify, and an implementation that skips that step is green against a genuine server and equally green against a forged one. Here the objection was not argued away, it was answered. The signature check is its own function with its own error code, so that "the step is missing" is not the same outcome as "the step passed", and there is a test that offers a forged signature and requires the login to fail. That test was proven by removing the check and watching it go red.

The crate the PostgreSQL check uses could not be reused, and that is measured rather than assumed. Its SCRAM implementation writes the client first message with an empty user name, because PostgreSQL sends the name in its startup packet instead. MongoDB has no such packet and reads the name out of that field. Patching the name in afterwards would change the very bytes that have already been signed over, and the server would reject our proof. There is no API to pass it.

A failure the protocol will not let you name

Now the part that no stub could have found.

RADIUS is checked with a Status-Server request, which asks the server whether it is alive and asks nothing else. The obvious alternative, an access request with a test account, was rejected: that is a real login attempt on the customer's network, with a real account somebody could use, writing a line into the customer's authentication log next to their real users. At a one minute interval that is fourteen hundred and forty lines a day that say nothing about their service, and on plenty of setups a wrong password is grounds for locking an account, which means the monitoring causes exactly the incident it was meant to see coming.

Then we measured what a wrong shared secret does, against a real server.

Nothing. A server that cannot verify the message authenticator on a request drops the packet, silently, with no error reply. That is in the specification, and it means the failure signature of a wrong secret is indistinguishable from a firewall rule, a wrong port or a dead machine.

You cannot fix that in an adapter, because it is a property of the protocol. What you can do is stop pretending otherwise. The "no reply" message names a wrong shared secret as a possible cause, in the text the customer reads, so that an administrator does not spend an afternoon in a firewall looking for something that is in a configuration file. And when the server does answer, the response authenticator is verified, and a mismatch there is reported as an authentication fact rather than a protocol error, because in that arrangement it genuinely is one.

That middle row is the only case where the difference is visible, and it is not the common one.

A server that cannot be measured, said out loud

The last measurement is the one we liked least.

SQL Server on Linux generates a certificate on first start. That certificate is X.509 version 1, a shape without extensions that has not been intended for TLS use for a long time, and the certificate parser underneath refuses it. So the encrypted mode fails with an unsupported certificate version.

It is worth being precise about why that mode still verifies anything. Identity is not weighed there: the name in the certificate is not checked. The signature is, because without it there is no handshake left, only an exchange of bytes with whoever answered. Weighing a signature requires reading the public key out of the certificate, and that is the step that refuses.

The outcome is therefore: a SQL Server using its own generated certificate cannot be measured by this check. The customer reads the real parser error as the detail, quoted rather than paraphrased, and the fix is a real certificate in the server configuration, which an administrator making a server monitorable tends to be doing anyway.

Two things were deliberately not done. The signature check is not skipped in that mode, because that would break the promise on precisely the setting most people pick and turn a visible refusal into an invisible weakness. And there is no dedicated error code for this case yet, because the only way to detect it would be to match on a library's human readable error text, which is an assumption about someone else's code that can shift without warning. That is a round of its own, with a test of its own.

There is also no mode without encryption at all, for a reason specific to this protocol: the password in the login packet is obscured with a fixed exclusive or and a nibble swap. That is an encoding, not encryption. Anyone who can see the bytes can read the password, so an unencrypted mode would send a customer's password across their network in clear text under a name that suggests otherwise.

What the canary found

All of it ran against real servers rather than stubs: fourteen checks against MongoDB bare and with TLS, a real RADIUS server, and a real SQL Server. The single most useful result is boring. A wrong password is an authentication failure everywhere, and never a protocol failure. That is the distinction that tells a customer whether to look at their credentials or at their network, and an earlier round had got it wrong in the PostgreSQL check by translating an error during the authentication step into "this is not a PostgreSQL server".

What we would write down

1. 01Count what a dependency holds, not just what it costs. A generic document type turns "we read four fields" from a property into an agreement.
2. 02A driver brings a query builder. If the promise is that no arbitrary query can be sent, the code to send one should not exist in the binary.
3. 03Two TLS stacks in one binary is a policy problem, not a size problem. The one you did not choose is the one you cannot patch.
4. 04In a hand written parser, the skip table is the dangerous part. An unknown type must be an error; skipping it by zero bytes shifts everything after it.
5. 05Accept a number in every width the protocol allows. One field arriving as a double is enough to read every failure as a success.
6. 06A fall back to a weaker mechanism must be a refusal, not a fall back. Green above a weaker login is worse than red.
7. 07When a protocol makes a failure indistinguishable, say so in the message. The wording is the only place that distinction can still be made.
8. 08Some servers cannot be measured, and that belongs in the documentation rather than in a workaround. The workaround here would have been to stop checking signatures.

For clarity about what this is: the three checks are built, tested against real servers, and behind a rollout switch. What remains unproven is the three of them travelling over the wire to a real customer agent, RADIUS accounting beyond the status request, and SQL Server in configurations other than a single instance on Linux.

From the platform work on Provibr. How the same canary approach exposed two defects in the earlier database checks is in a canary that runs against real servers, and how the credentials these checks use reach the agent is in a secret that travels once.