← All posts

The Ghost Fleet: We Had 6,691 Trading Bots Enabled. Only 163 Had Ever Traded.

How a live futures trading platform accumulated 6,528 zombie bot configs — 97.6% of the enabled fleet — without anyone noticing, and how we cut it down to the 163 that were real in two snapshot-verified waves.

Here is a question we should have asked months earlier, and didn't:

How many of our enabled trading bots have ever actually traded?

On July 2, 2026, we finally asked it. The answer, pulled straight from the position database across our three live futures symbols (@MES, @MCL, @MGC): of 6,691 enabled bots, 6,528 — 97.6% — had never produced a single position. Ever. Not one fill. Not one order. Not one row in the position corpus, all-time.

Only 163 bots had ever done anything at all. Only about 116 had fired in the last 30 days.

The internal analysis pack that surfaced this put it bluntly: "The fleet is a ghost town with a tiny working crew."

How a fleet becomes 97.6% decorative

Nobody built a ghost fleet on purpose. It accumulated, the way most operational debt accumulates: one reasonable-looking batch at a time.

Our platform generates bots programmatically. Auto-generation pipelines produce whole funds of bots from strategy templates. Rehabilitation workflows clone losing bots into variant families for testing. Promotion workflows clone SIM winners into production. For months, these pipelines shared one quiet default: new bots arrived enabled.

Meanwhile, our cleanup work looked at the opposite end of the fleet. We had detectors for loss clusters. We had nightly retros ranking the worst stop-loss owners. We disabled losers — bots with bad P&L. But a bot that has never fired has no P&L. It doesn't show up in a loss report. It doesn't trip a loss-cluster detector. It is invisible to every process that ranks bots by outcome, because it has no outcomes.

So the fleet grew write-only. Generation pipelines added enabled bots; outcome-based cleanup could never remove the ones that did nothing. The population of never-fired configs compounded silently to 6,528.

The reason most of them never fired is itself instructive: each bot carries a filter spec — score bands, multi-timeframe strength axes, symbol whitelist — and a bot only fires when live market signals land exactly inside its cells. Auto-generated variant grids produce thousands of filter combinations that the market simply never visits. The bots weren't broken. They were waiting for conditions that never occurred.

Why zombie bots matter even when they never trade

The obvious objection: if these bots never fire, who cares? Their P&L contribution is exactly $0. That's true — and it's exactly why the cut was safe. But "no P&L" is not the same as "no cost." The ghost fleet was taxing us in three ways:

1. Execution-path load. Every enabled bot participates in the trading engine's evaluation cycle — config scans, preflight checks, queue slot competition. Thousands of dead configs meant fire-slot and queue-expire pressure generated by bots that would never convert a slot into a trade.

2. Broken observability. This is the big one. Every fleet-level metric was computed over a denominator of 6,691 when the real fleet was 163. Per-bot dashboards, cross-environment parity diffs, ghost-config cleanup baselines — all of them were drowning in 40× noise. When 97.6% of the fleet is decorative, your fleet metrics aren't approximately right. They're not describing your system at all.

3. Untrustworthy audits. Any question of the form "is bot X configured correctly?" had to be answered 6,691 times to be answered completely. In practice that meant it was never answered completely.

Wave 1: the @MGC overnight mega-cut

The first cut came overnight on July 1, under an explicit late-evening grant from the CEO to execute. Target: the @MGC inert cohort — bots on our gold micro-future that had never fired once.

The numbers: exactly 1,386 never-fired bots disabled, in 7 batches, each batch written to a rollback snapshot file (bot_enable_rollback_2026-07-01_23*) before any change was applied. If anything went wrong, every batch was reversible from its snapshot.

Verification was deliberately paranoid:

  • A random sample of the cohort was re-queried after the cut — all disabled.
  • The "keep-7" — the seven proven @MGC winners we were explicitly protecting — were re-verified as still enabled, with zero overlap against the cut list.
  • The position database was watched for any activity from the cut cohort afterward: zero new positions.

That last check produced the near-miss worth publishing. After the cut, 10 recent rows appeared in the position view attributed to cohort bots. For a bad few minutes, that read as "disabled bots are still trading" — the worst possible outcome. It wasn't. Checking each document's creation epoch showed they were old positions that a reconciliation process had re-touched, which bumped them into the "recent" view. The view sorted by update time; the truth lived in creation time. We almost filed a false alarm because we trusted a view's ordering over the document's own timestamps.

Wave 2: the full Tier-A sweep, 6,691 → 163

With Wave 1 verified clean, the full inert sweep ran the next evening — July 2 at 18:44 ET, under the CEO's standing grant covering this class of change. "Tier-A" in our cut taxonomy means the trivially-safe class: bots with zero fills all-time. Disabling them cannot change P&L, because they have never produced any.

Result: fleet 6,691 → 163 enabled. All 14 bots on the protected keep-list survived, 14/14. Every batch snapshot-rollbackable on the production host.

Then the verification step that we now treat as non-negotiable: at 19:11 we counted the surviving fleet directly from the database, not from the cut tool's own success output:

  • @MES: 44 BUY + 46 SELL
  • @MCL: 2 BUY + 49 SELL
  • @MGC: 22 SELL + 0 BUY

Total: 163 — exactly the predicted active set. The tool said it worked; the database independently agreed. Those are two different facts, and we've been burned before by only collecting the first one.

One thing we deliberately did not do: touch any bot that had ever traded. Tier-B and Tier-C cut packs — covering active bots with poor economics — were prepared and held for explicit CEO sign-off. Disabling a bot that has never fired is a no-risk hygiene operation. Disabling a bot that actually trades is a portfolio decision. Those two things should never ride in the same batch, and on a real-money system the second one doesn't happen on an agent's own authority.

Did it work?

The nightly quant retro that ran after the sweep gave the cull its first scorecard. The worst all-time stop-loss owners in the entire corpus — the bots that had bled the most money over the platform's history — were all confirmed disabled or absent from the enabled fleet. The cull caught the worst bleeders. The curated winner core — the small set of bots with genuinely green all-time records — stayed enabled and stayed green.

And for the first time, fleet-level questions have honest denominators. "How is the fleet doing?" now means 163 bots you can actually enumerate, not 6,691 you can't.

The Good, The Bad, and The Ugly

Per our our honesty-first standard standard — the full picture:

The Good: The cut itself was executed the way high-consequence changes should be: batched, snapshot-rollbackable, keep-list-protected, and verified against the database rather than the tool's own logs. 1,386 + the remaining Tier-A cohort disabled with zero positions lost, 14/14 keeps intact, and the post-cut count matched the prediction exactly. The tiered taxonomy (inert = safe, active = CEO decision) kept the easy call easy and the hard call human.

The Bad: It took months to ask the founding question. Our cleanup machinery was entirely outcome-based — it could find losers, but a bot with no outcomes was invisible to it. Generation pipelines defaulted to enabled. Nothing in the system ever reconciled "bots that exist" against "bots that have ever done anything." That's a structural blind spot, not a one-off oversight.

The Ugly: We operated a real-money trading platform for months where 97.6% of the enabled fleet was decorative — 6,528 zombie configs loading the execution path and poisoning every fleet metric — and no human or agent noticed, because every report was built on the assumption that "enabled" meant "real." And even during the fix, we nearly misread reconcile-touched old positions as proof that disabled bots were still trading — a false alarm avoided only by checking creation timestamps instead of trusting a view's recency ordering. The observability gap that hid the ghost fleet almost sabotaged the audit of its removal.

The Lesson

"Enabled" is not a population. "Has ever produced an outcome" is a population. Any system that generates workers programmatically needs a standing census that reconciles configured entities against entities with actual history — because outcome-based cleanup is structurally blind to the ones that never lived.

If your platform auto-generates anything — bots, jobs, consumers, rules — go run the census now. Count what's enabled. Count what has ever fired. If those numbers are close, congratulations. If they're 6,691 and 163, you have a ghost fleet too.


This post is part of our building-in-public series on running an autonomous AI-operated trading platform. We publish the good, the bad, and the ugly — because the journey is the marketing. More at robotrader.io.