A Bigger Pile Is Not the Same Pile

Our two trading environments held about 11,800 and about 10,500 bot configurations. The bigger one obviously wasn't missing anything — so the gap had to be a configuration problem, not a content problem. That reasoning was wrong, and the number that proved it wrong took ten minutes to compute. Count equality is not set equality, and almost every parity check we've ever seen gets this backwards.

We run two environments for our automated trading platform: a simulation environment where strategies are validated, and a live one where they trade real money. Keeping them in agreement is not a nice-to-have — the entire value of the simulation environment is that it's a faithful model of the live one. When it drifts, every result it produces is quietly worth less.

So we audited. The question was simple: does the simulation environment have everything the live one has?

The first answer took about a minute to produce, and it was reassuring.

The simulation environment had more. Whatever was wrong, it wasn't that things were missing — you can't be short on inventory while holding thirteen hundred more units than the other warehouse. The conclusion followed naturally: the gap is an enablement problem. The configurations are all there, most of them are just switched off. That's a cheap fix — flip a flag on the right subset and the environments match.

That framing was reported, and it was wrong. Not wrong in its arithmetic — both numbers were correct — wrong in what it was permitted to conclude from them.

The number that broke it

The agent that produced the count did something that deserves more credit than the finding itself: it went back and checked its own framing before anyone acted on it.

Instead of comparing how many configurations each environment had, it compared which configurations each environment had — by computing a content fingerprint for each one and comparing the resulting sets. Not by name, not by identifier, not by count. By what each configuration actually contained.

The result:

By the time of this audit the live environment held roughly 9,200 distinct enabled configurations — a number that deserves a note, because it is not the one that appears in the previous post in this series. That one ends with a fleet culled to 163. In between came a deliberate, CEO-directed fleet-wide re-enablement: having established which configurations were genuinely inert, we turned the surviving population back on at scale. The cull and the re-enablement are the same programme, a month apart.

So, against that ~9,200:

• About 5,700 of them existed in the simulation environment at all — of which a few dozen were already switched on, leaving roughly 5,600 to enable.
• Which left roughly 3,500 that did not exist there in any form.

Not disabled. Not renamed. Absent. Meanwhile the simulation environment's larger total was padded with thousands of configurations the live environment had never had — experiments, variants, and generated combinations that had accumulated on the sim side and never crossed over.

Two piles. One bigger. Barely overlapping in the way that mattered.

The agent's own summary of its error is the best sentence anyone wrote that day:

"My earlier framing was half right — it came from raw counts, and a bigger pile is not the same pile; only fingerprints settle it."

Why the wrong answer was so convincing

It's worth sitting with how reasonable the first conclusion was, because that's what makes this pattern dangerous rather than merely embarrassing.

The counts were accurate. The comparison was arithmetically valid. The inference — more items means nothing is missing — is the kind of thing that feels like it doesn't need checking. And it's true of piles of identical objects. Thirteen hundred extra screws means you're not short on screws.

It is false the instant the objects are distinguishable, and configurations are the most distinguishable objects in our system. Each one is a specific combination of filters, sizing, and behaviour. A configuration is not a unit of inventory; it's an identity. And set membership tells you nothing about set size, in either direction.

The failure mode has a shape worth naming: an aggregate is a lossy summary, and comparing two lossy summaries tells you about the summaries, not the things. Any time you compare two systems by a count, a total, an average, or a percentage, you're comparing shadows. The shadows can match perfectly while the objects casting them have nothing in common.

What it would have cost

The wrong framing had a concrete action attached, which is what makes it more than a semantic quibble. "It's an enablement problem" implies a specific, cheap remedy: flip flags on the existing configurations until the environments agree.

Had we done that, we'd have produced an environment that reported parity — matching enabled counts, a satisfying number on a dashboard — while thousands of the live environment's actual configurations still had no counterpart in simulation at all. Every subsequent validation run would have been performed against a model that was silently missing a third of what it was supposed to model, and it would have looked more correct than before we started.

That's the real cost of a count-based parity check: it doesn't just fail to find the gap, it manufactures evidence that the gap is closed.

The corrected plan split into two phases with very different weights: enable the roughly 5,600 configurations that genuinely existed and were merely switched off (the few dozen already enabled needed nothing) — cheap, reversible, low risk — and separately create the roughly 3,500 that were genuinely absent, which is a large write and a much more serious piece of work. The first phase ran. The second was deliberately not started on the same authority.

The verification we now insist on

The first phase completed and reported success. We didn't accept that.

Instead of reading the tool's own log, we re-ran the check against a sample of the affected configurations in dry-run mode and asked a different question: how many still need changing? The answer came back zero, with every sampled item reporting as already in the desired state. The writes had genuinely landed.

That's the same discipline that showed up in our fleet-cleanup work last month, and we've made it standard: the tool's success message and the system's actual state are two different claims, and only the second one is evidence. A log line saying "5,623 updated" is a report of intent. A fresh query showing nothing left to update is a report of reality.

The good

The error was caught by the agent that made it, before anyone acted on it, and stated plainly rather than quietly amended. "My earlier framing was half right" is a more useful sentence than any silent correction would have been, and it's the reason the wrong plan never ran.

The gate also held. The corrected analysis produced two phases with very different risk profiles, and the large, hard-to-reverse one didn't proceed just because the small one had gone well. Momentum is not authorisation.

The bad

The count-based framing was reported before it was checked. It sat as the working understanding for about two hours, and a plan was built on top of it. Two hours is not a disaster, but the only reason it wasn't longer is that the same agent chose to re-examine its own conclusion — not that anything in the system flagged it.

We also don't have an automated parity check that works on identity rather than counts. We have the capability — that's how the correction was computed — but nothing runs it on a schedule and alerts when the sets diverge. The audit happened because someone asked, and the correction happened because someone doubted. Neither is a control.

The ugly

We have been comparing these two environments for months. Every one of those earlier comparisons used counts.

We don't know how long the roughly 3,500 configurations were absent, because nothing was ever measuring the thing that would have revealed it. The gap wasn't discovered because it grew large enough to notice — it was discovered because someone finally asked a different question. It could have been that size for a very long time, and every "the environments look aligned" statement we made in that period was made with an instrument that could not have detected otherwise.

That's the uncomfortable generalisation, and it's not really about trading: you can only find the drift your instrument is shaped to see. A count-based check will report healthy across an unbounded amount of divergence, forever, and it will do it with a perfectly accurate number.

What we'd tell another team

1. Compare identities, not cardinalities. If your parity check produces a number rather than a set difference, it cannot tell you what's missing — only that something might be.
2. Treat "more than" as meaning nothing about coverage. The bigger side of an unequal comparison is the one most likely to lull you, because "we have extra" reads as surplus rather than as different.
3. Watch for aggregates in incident framing. Counts, totals, and percentages are the natural language of a status report, and every one of them is a lossy summary standing in for the thing you actually care about.
4. Verify writes by re-querying, not by reading the writer's log. Intent and outcome are separate claims.
5. When a cheap explanation and an expensive one both fit the data, notice which one you'd prefer. "It's just an enablement problem" was the cheaper story, and it was the one that came first.

Two numbers, thirteen hundred apart, both accurate, and they described sets that differed by thousands. The arithmetic was never the problem. The question was.

Part of an ongoing series on what actually happens when you automate trading with AI agents — published including the parts that don't flatter us.