Skip to content
Thomas Fairhurst
Chapters

Chapter 04 · Orchestration

Nine ways to organise my agents

I improved the way my agents were organised the same way I improved the bot. In 23 days I ran nine designs, measured each one, and carried what it taught me into the next.

9designs, 22 Oct to 13 Nov
138versions the agents uploaded to the ladder
7,720agent threads started, 26 Oct to 13 Nov

Nine designs, each measured

Each design ran from 8.7 hours to six days. The large dot is the agent in charge; the dots under it sketch how the rest were arranged, not a head count.

22–25 OctWaves behind a gate

one orchestrator, waves of 14 agents

71 hran
25–31 OctA learned-bot rebuild

one orchestrator, 631 sub-agents, GPU training

6 daysran
31 Oct–4 NovProofs and review rounds

builders, each with fresh reviewers

4 daysran
5–6 NovA coordinator and 30 readers

readers study losses, the coordinator fixes

26 hran
7–9 NovFix agents behind a gate

one lost game each, automatic uploads

64 hran
9–10 NovDirector, leads, workers

a director over 14 leads

8.7 hran
10 NovOne agent

with up to 10 helpers, then alone

19 hran
11–12 NovBets

every change a written prediction

41 hran
12–13 NovOne lost game at a time

replay it until it is won

23 hran

What each design showed, and what I changed

DesignWhat it showedWhat I did next
Waves39 uploads, #347 to #4 on the ladderBet on a learned bot to go past hand-written rules
Learned botIts best won 12 of 140 games against v50Deleted the repo and started again
ProofsFour days without a version that beat v50Cut reviews to three rounds, then none, and began a new bot
ReadersThe win rate against v50 rose, then fellDesigned a gate that judges each fix on its own
Fix agents73 uploads, 2,258 to 2,477 (#12), then gains flattenedGave the loop to a director, to find bigger changes
Leads3 uploads, about +69; six pieces of play built one at a time all read zero or worseOne agent doing everything, so the ideas connect
One agent14 uploads, about +104, all before it went aloneEvery change a written bet; the ladder a scoreboard, never a test
Bets9 uploads, about +48Deleted the loop and pointed every hour at the bot's code
One game8 of 10 set games won, but only level with v139 on new mapsUploaded nothing: v139 stayed live
The “about” gains chain each version’s own platform reading, and each reading carries ±40 to ±150, so the chains cross zero.

Reviews cost more than they found

The third design made every plan pass rounds of fresh reviewers until a round found nothing. Measured on 3 Nov, it cost far more than it found, so I cut it to three rounds on 4 Nov, to none by default on 5 Nov, and deleted the plans with it.

Highest review round each document reached

Plan for the v51 loop
37
Register of axioms and facts
31
Upload rules
28
Upload model
24
52 of 63.5agent-hours in commands on 3 Nov spent waiting for GPT reviewers
21 of 28forced re-reviews that found nothing
353klines deleted on 5 and 6 Nov, when the plans went

Readers find the cause, a gate judges the fix

From 5 Nov one coordinator wrote every fix, fed by about 30 reader agents at a time, each studying one lost game: 1,301 readings in 26 hours, a median 18 minutes each.

  • Bulk games against old versions
  • One reader per lost game
  • The coordinator writes the fixes
  • The next bulk run judges them

The reader’s model was chosen by a test, on lost games whose cause was already known.

Tries that found the cause of a test loss

GPT 6.1 Soltest game 1
6 of 6
GPT 6.1 Soltest game 2
4 of 5
Opus 5.5 mediumtest game 1
0 of 4

Games won against v50

5.5%after the 3rd round of bulk games, 5 Nov 19:45
20%9 hours later
12.5–19.5%after rounds 72 to 74, 6 Nov 20:55

When it plateaued I stopped it. The coordinator landed four or five fixes a round and judged each on a single game, so I designed a gate: a branch and its parent play the same maps from the same seats, a game whose result changes is a “flip”, and a statistical test on the flips decides.

From a lost game to a shipped fix

From 7 Nov the coordinator wrote no code. Each lost online game went to a fresh fix agent, and a fix that passed the gate was merged with others and uploaded with no human step.

  • A lost online game
  • A fix agent on its own branch
  • Watch it, explain it, count it
  • Fix, scenario test, fuel proof
  • Gate against its parent
  • Batch merged and uploaded

Two cycles, as they happened

2 hours: v59

  1. Two online losses on one mapTo Mite and Zarzorki.
  2. A fix agent startsOur champion stood beside a one-tile monolith while another unit took the 200 gems.
  3. Gate fails−0.34 points over 40,528 maps. The fix made a unit step on and off in a loop.
  4. A fresh agent narrows the rule
  5. Gate passes+0.21 points over 273,104 maps.
  6. Live as v59+0.6 points over v58.

6 hours: v42

  1. A ranked loss to MiteMite cancels a tribute, strikes, and restarts the tribute in the same turn.
  2. Nobody gates itv50, the gate's opponent, never does this. Two agents stop: “too rare”.
  3. I step in“We fix the exploit, exploit back if it's general.”
  4. Agent 4 passes the gate+1.06 over 10,372 maps, from a change it found on the way: going home now uses this game's death rate.
  5. Ships inside v42Six hours from the loss to the ladder.

It climbed fast, then flattened, and the gate’s promises shrank when replayed. To me that meant small fixes had run out, so on 9 Nov I gave the loop to a director to look for bigger changes.

+21 ± 7Elo an hour against the field on the evening of 7 Nov
+0.9 ± 2.4Elo an hour on 8 Nov
~41%of a passed fix's claimed gain that showed up when replayed

Measuring the agents, not only the bot

Twice I had a design measured before replacing it. On 9 Nov a redesign I ran from my laptop measured the coordinator loop of 5 to 9 Nov.

846times it was woken; 68% of them decided nothing
3.08 billiontokens of its own context it re-read
44context compactions, then 219 memory edits to rebuild what it knew
43%of agent time spent blocked on local jobs
These counts survive only in the 9 Nov analysis; the logs behind them are gone.

On 11 Nov my own advisor thread ran 20 reviewers over the director’s design. Six read “Broken”. One measured the night the director had just spent, with helpers.

The director’s night, measured

MeasureReading
Uploads in 12 h14, and 13 were replaced before 1,200 games
Silent game queuethe queue of practice games stopped for 3 h 13 min unnoticed: about 1,400 games lost
One-call turns92.9% of the director's 4,056 turns made one tool call
Working filerewritten 114 times, 97 lines grown to 754
Helpers78 runs, 59.5 agent-hours, 11.0M output tokens: 2.8 times the director's own
Helper code32 of 50 runs that wrote code never reached the main branch

Bets, then one game at a time

From 11 Nov every change was a “bet”: a written prediction, a number showing it happens in games, and a judge. A hook refused remote games run without one.

What happened to the 85 bets

Kept
9
Refuted
5
Closed untested
31
Still open at the end
40

The kept bets added about +20 Elo on our own league in two days. On 12 Nov I had the loop deleted and put all the time on the bot: ten lost games, each replayed exactly, fixed one at a time until won, keeping every earlier game won.

“Fix the bot until this one game is won.”
12 Nov, 23:32

Fitting ten games, checked on new maps

Games won of 600 on maps it had never seen
294
115
  • Before
  • After 321 commits

The check on unseen maps caught the overfitting before anything shipped. Repairs, by deleting rules, brought the branch back level with v139, 609 of 1,200 games, so v139 stayed live. The lesson the agents wrote down: check every rule on maps it has never seen.