Our Proof of Life Was the Leak
For about twenty days this month, one of our servers answered requests for a hostname nobody had claimed, and what it answered with was a database's welcome page — including the name and version of the software running it.
Nothing was taken. Every request that arrived asking for something real got a 404. No data was exposed, no account was touched, and this was not a breach. What was exposed was a banner: a small, cheerful string saying here is what I am and here is which version I am.
That is the boring part. Here is the part worth publishing.
The default was never decided by anyone
Our web server had a list of site configurations, and not one of them declared itself the default. So when a request arrived for a hostname that matched nothing, the server fell back to the first configuration in alphabetical filename order — and the file that happened to sort first was the one for a database.
Nobody chose that. There is no decision, no ticket, no review. A filename sorted earlier than another filename, and that determined what an unclaimed hostname served to the public internet for twenty days.
This is the most ordinary kind of production defect there is, and it is invisible to every review that asks "is this configuration correct?" — because each configuration was correct. The defect lived in the space between them, in a default nobody wrote.
🔻 The part that should worry you: the monitor was green because of the leak
Here is what took this from an untidy default to a genuinely uncomfortable lesson.
Our load balancer's proof that the server was alive was the banner itself.
The health check asked the server whether it was up. The server replied with the database's welcome page. The health check saw a valid response and marked the host healthy. The leaking string was functioning as our liveness signal.
Which produced this: at one point during the incident, a completely healthy host was pulled out of rotation for two hours — because the thing the load balancer had been using as evidence of life stopped looking the way it expected. The host was fine. The proof was what broke.
We had built a monitor that was green because of the bug, and which went red when the bug was disturbed. It was not measuring health. It was measuring the leak.
A green tick is not evidence until you know what the instrument is actually reading. Ours was reading the wrong thing for long enough that fixing the wrong thing looked like an outage.
The good
When it was finally looked at properly, it was found and fixed the same day — and the fix carried something better than a patch.
Two competing explanations had been on the record for weeks, and nobody could adjudicate between them, because each required visibility into a system the other party could not see. One said the traffic reached our web server and was being mishandled there. The other said the traffic never reached it at all.
The fix settled the argument by landing. A change made at the web-server layer cannot remove a banner from traffic that never arrives at the web server. The banner went away. Therefore the traffic was arriving — which retired one explanation and confirmed the other, without a single additional test.
That is a satisfying kind of evidence: not a measurement, but a consequence that only one hypothesis permits.
The bad
Twenty days. Not twenty minutes. The condition was live, publicly reachable, and unnoticed for nearly three weeks, on a system that is monitored continuously by an organisation whose entire claim is that it watches itself.
And we were self-certifying against an instrument that could not see the problem. An earlier check had confirmed the configuration was serving the right thing — but the field it inspected reports the name of the configuration that matched, not the name that was requested. It could never have detected this. It was not wrong; it was answering a different question, confidently.
The ugly
We still cannot prove the class is closed, and we are not going to say we can.
What was fixed is the mechanism: the default is now an explicit, deliberate stub instead of an accident of alphabetical order. Any hostname invented tomorrow lands on something chosen on purpose. That is the real fix and it is durable.
But "the banner is gone from these names" and "nothing else can reach this system by another route" are different claims, and we have only established the first. Our own closure test — the one that would have measured the second — turned out to be unusable: the logs do not record where a request came from, only which internal component forwarded it. The test could not grade the thing it existed to grade.
That is the second instrument in this story that was confidently answering the wrong question. We found it while trying to prove we were finished, which is exactly the moment you least want to discover it — and exactly the moment you are most tempted not to look.
What we changed
The default is explicit. The health check will not be allowed to treat an arbitrary valid response as proof of life. And the standing rule that came out of it, which applies well beyond this incident:
Before you trust a green signal, ask what the instrument is reading — and whether it could physically observe the failure you are asking it about. A check that cannot see the fault will report success forever, and it will be telling the truth about the wrong thing.
We publish these because the alternative is a company that appears never to make mistakes, which would tell you only that we do not write them down.
RoboTrader trades CME micro futures. Trading involves substantial risk of loss and is not suitable for every investor. Past performance is not indicative of future results.