Two rules, and a dead link between them

We crawled the production site properly for the first time in a while: every internal link followed, every declared language alternate checked, every structured data block validated, every image checked for alternative text.

1776 pages, 1750 sitemap entries, and exactly one defect. Zero orphans, zero redirect chains, zero language alternate errors over 17218 declarations, zero invalid structured data blocks over 1642, zero images without alternative text over 15463, and 920 social preview images all answering 200.

The one defect was 18 URLs returning 404: the documentation pages of two billing panels, in all nine languages. Each of those 18 had exactly one internal link pointing at it, from the integration page for the same product in the same language.

Both halves of that are on purpose. That is what makes it worth writing down.

The two rules

Some of our integrations sit behind a rollout switch. When the switch is off, the provider disappears from the catalogue, from the sitemap, from the counters and from the type selector in the panel, all derived from one table so there is no second list to forget.

Two of them are not quite ordinary. For FOSSBilling and Paymenter there is a module you install in their panel, and the page at /integrations/fossbilling describes that module. It is not a page about a connection we offer; it is a page about a thing you can download today. So there is a rule that keeps those two integration pages standing when the provider behind the switch is hidden, and it carries its reason in the source.

The documentation page is the opposite case. /docs/fossbilling describes the connection, which is exactly what the switch turns off, so it disappears with the provider.

Both rules are right. One slug, two surfaces, two answers, and the code has two functions rather than one that would be wrong half the time.

What nobody had checked is what happens between them. The surviving page carried the topical link to the vanished one.

The rule was already written down

The function that produces that link resolves a documentation path for an integration slug. Its own doc comment said, before any of this happened, that a link to a guide that does not exist is not a topical link but a dead one.

The rule was there. It was measured against the wrong list: the set of documentation pages that exist in the source, rather than the set that answers in this build. Those two are the same set almost all the time, and the day they differ is the day something is hidden.

The repair is one line. That is not the interesting part; the interesting part is that a correct rule, written out at length in the place it belongs, is not a control. Prose in a comment describes an intention. Only something that runs can hold you to it.

A third case no crawl could have found

We wrote a guard rather than just fixing the line: for every integration slug, the resolved documentation path must be one this build actually serves. It was proven red first, and it reported the two pages the crawl had found.

It also reported a third: our backup server integration, which is behind a different switch. Its documentation page is hidden, and the link to it was built exactly the same way.

The reason no crawler will ever find that one is that its integration page is hidden too. The dead link lived on a page that returns 404 itself, so there is no path from the outside to the thing that is wrong. It would have surfaced on the day somebody turned that switch on and shipped a page with a broken link on it, and the two events would have been weeks apart.

A crawler can only find defects on reachable pages. That sounds obvious written down and is easy to forget when a crawl comes back clean, because a clean crawl feels like a statement about the code rather than about the pages that happened to be visible.

The guard has one more test that does nothing except keep the first one honest: it asserts that more than ten slugs really do resolve to a documentation page. Without it, a change that made the resolver return nothing for everything would produce no dead links and no topical links at all, and those two read identically in a green test run.

The other half: telling a search engine something changed

The same round added a submission channel, which is a small thing with one surprise in it.

The protocol is one POST that tells several search engines a URL has changed. Ownership of the host is proven by serving a key as a text file at the root. Our key comes from the environment, and when it is absent the whole thing does nothing and says so once. That is the normal state in development and in every test.

Two details are worth repeating because both are traps.

The key file is served through a rewrite, because a dynamic route cannot sit at the root next to the language segment. A rewrite that matches anything dot txt at the root would also swallow the robots file. Two things stop it: the rewrite runs in the phase that only sees requests no real file matched, and the key pattern requires at least eight characters, while robots is six and llms four. The lower bound is not tidiness; it is the second lock.

And the submission hangs off the function that already knows public content changed, not off the deploy. A release that only touches a panel screen would otherwise announce 1750 addresses as modified every time, which is a good way to be ignored. When the price table changes, the two routes that render prices are submitted in nine languages, which is 18 addresses.

The thing that looked like a misconfiguration

The first real submission after the deploy, with the production key in place, returned 403.

The key file returned 200 from everywhere we tried, including with a crawler user agent. Every ingredient checked out. The natural next step is to go and look at the edge configuration, or the rewrite, or the phase ordering, which is where an hour goes.

What we did instead was try two smaller things. A single URL submission returned 200 immediately. The same bulk submission, one minute later, also returned 200, at both endpoints.

So the 403 was a first contact effect on their side, not a defect on ours. The key file had not yet been fetched and cached by the receiving service at the moment the first bulk request arrived.

We have written that down where the runbook lives, because the failure is perfectly designed to send somebody hunting: the status code says forbidden, which is a statement about permission, and everything about permission was correct. Before investigating an authorisation failure on a first request, make the same request again. It costs nothing and it is the cheapest hypothesis in the list.

There is a related note for whoever rotates that key: the file is served with an hour of caching, so a rotation needs a purge or an hour of patience, and during that window the same 403 means something completely different.

What we would write down

1. 01Two correct rules can produce a defect in the seam between them. Neither owner is wrong and neither is looking.
2. 02The same identifier can have two surfaces with two right answers. Two functions, not one.
3. 03A rule in a comment is not a control. Ours was written out and measured against the wrong list.
4. 04What exists and what answers are different sets, and they diverge exactly when something is hidden.
5. 05A crawler only finds defects on reachable pages. Ours could not see the third case at all.
6. 06A guard finds the instance that has not shipped yet. That is the whole reason to write one instead of fixing the line.
7. 07Give every guard a test that proves it is looking. No dead links and no links at all read the same.
8. 08A lower bound in a pattern can be a security boundary. Eight characters is what keeps a rewrite off the robots file.
9. 09Announce content changes from the thing that knows content changed, never from the deploy.
10. 10Retry before you diagnose a first request failure. A forbidden that becomes a success one minute later was never about permission.

For clarity about what this is: the crawl covered 1776 pages against production and 1750 sitemap entries with 913 modification dates none of which were in the future, the fix is a single line guarded by three tests proven red first, the submission path was exercised end to end against a stubbed endpoint including five failure paths, and the sweep over a standalone production build finished at 96 green. Not proven: at the time the fix was written it was green on a local production build and had not yet been confirmed on production, and two route families in the crawl produced no pages at all because their registers are empty.

From the platform work on Provibr. What a billing migration can and cannot carry is in fifteen things a billing migration cannot carry, a deploy whose schema step never ran is in the migration that never ran, and another check that was green because it never arrived is in green because it could not get there.