Skip to content
Thomas Fairhurst
Chapters

Chapter 13 · Verdict

What worked, and what I’d do differently

Five decisions carried the bot to #7. My costliest mistakes were not bugs. Three times I judged the bot on the wrong thing, and each one showed me how a test should be built.

+333rating in 70 hours once I rebuilt the bot and deleted what nothing used
12 of 140games the best learned bot won against my rules bot
0versions after v139 that read better, so none went up

What worked

  1. Hill-climb a rules bot behind a testEach version went live only once it passed the test. v50 reached #2 and became the yardstick for everything after.
  2. Fast games on Imperial's machinesA native build of the game engine: 3,600 games, decoded into tables, in 170 s.
  3. Rebuild as a new bot, delete everything unused+333 rating in 70 h, 8 to 11 Nov.
  4. Judge against the finalists, 1,200 live games a versionThe 9 ideas I kept came from it; v136 measured +48 on the ladder.
  5. Never test on the live ladderIt kept v139 live while the last day's branch was worse.

What I’d do differently

Three of these four are about what the bot was judged on.

  1. Improve the bot that wins while trying a new ideaSix days on learned and reinforcement-learning bots: the best won 12 of 140 games against my rules bot v50. Meanwhile v50's rank slid from #2 to #16.
  2. Build a working bot before proving itFormal proofs in Lean, a belief-annotated v50, long review rounds and lists of planted bugs: all deleted. The bot that climbed began only on 4 Nov.
  3. Test against the field, not against myselfA test that played my own bot, one small fix at a time, claimed +272 Elo. The ladder showed +55 ± 48.
  4. Check every change on maps it has never seenFixing ten lost games one by one fitted the bot to those ten. About 17 h passed before the check that showed it.

A test that played only my own bot

From 7 to 9 Nov a fix shipped if it did better than its parent in games against my own bots. Most of what it gained there did not carry over to other teams.

What the fixes gained

Elo, the loop's last 27 hours
+272
+55 ± 48
  • The test claimed
  • The ladder measured

What the test missed, watching one game found: 35 of the bot’s 114 game calls were never used, every read of enemy wards among them.

“We are basically playing with a blindfold”
9 Nov, after watching one game

The last day, fitted to ten games

The method was mine: take ten lost games, each one map, one side and one fixed opponent, so it replays the same every time, and fix the bot until each is won.

  • Ten lost games
  • Fix the first until it is won
  • Every earlier game still won
  • Next game

Wins on unseen maps, before and after 321 commits

Of 150 games
78
22
Of 600 games
294
115
  • Before
  • After

Each rule to blame had been built from one sight in one game of the set. Deleting those rules brought the branch back, but only to level with v139.

8 of 10set games won at the end
609 of 1,200games the repaired branch won against v139
0versions uploaded from the last day
“Do we have anything better than v139?”
13 Nov, 22:45. The answer was no.

How I’d judge a change next time

That evening my agent wrote down how a rule should have been judged. It was never applied, and it is where I would start.

600+unseen-map games600 to 1,200, the rule against its parent
z ≥ 2to keep itanything weaker is inside the noise
Censusfor farming rulescount where the gems go, not only wins
1st splitto read a lost gameonly where the two builds first play differently

What my agents wrote down

My agents kept a folder of notes to their future selves. These are the lessons in it.

On judging

  • Check every rule on unseen maps
  • 150 games are noisy by about 6 to 9 wins
  • Keep a rule only at z ≥ 2 on 600 to 1,200 games
  • Switch rules off in bundles
  • Count where the gems go, not just wins
  • Self-play misread fight and gold changes

My working rules, as they kept them

  • Never wait idle
  • Time is shorter than it feels
  • Usage is not a worry
  • Take everything a written rule allows