What worked
- Hill-climb a rules bot behind a testEach version went live only once it passed the test. v50 reached #2 and became the yardstick for everything after.
- Fast games on Imperial's machinesA native build of the game engine: 3,600 games, decoded into tables, in 170 s.
- Rebuild as a new bot, delete everything unused+333 rating in 70 h, 8 to 11 Nov.
- Judge against the finalists, 1,200 live games a versionThe 9 ideas I kept came from it; v136 measured +48 on the ladder.
- Never test on the live ladderIt kept v139 live while the last day's branch was worse.
What I’d do differently
Three of these four are about what the bot was judged on.
- Improve the bot that wins while trying a new ideaSix days on learned and reinforcement-learning bots: the best won 12 of 140 games against my rules bot v50. Meanwhile v50's rank slid from #2 to #16.
- Build a working bot before proving itFormal proofs in Lean, a belief-annotated v50, long review rounds and lists of planted bugs: all deleted. The bot that climbed began only on 4 Nov.
- Test against the field, not against myselfA test that played my own bot, one small fix at a time, claimed +272 Elo. The ladder showed +55 ± 48.
- Check every change on maps it has never seenFixing ten lost games one by one fitted the bot to those ten. About 17 h passed before the check that showed it.
A test that played only my own bot
From 7 to 9 Nov a fix shipped if it did better than its parent in games against my own bots. Most of what it gained there did not carry over to other teams.
What the fixes gained
- The test claimed
- The ladder measured
What the test missed, watching one game found: 35 of the bot’s 114 game calls were never used, every read of enemy wards among them.
“We are basically playing with a blindfold”
The last day, fitted to ten games
The method was mine: take ten lost games, each one map, one side and one fixed opponent, so it replays the same every time, and fix the bot until each is won.
- Ten lost games
- Fix the first until it is won
- Every earlier game still won
- Next game
Wins on unseen maps, before and after 321 commits
- Before
- After
Each rule to blame had been built from one sight in one game of the set. Deleting those rules brought the branch back, but only to level with v139.
“Do we have anything better than v139?”
How I’d judge a change next time
That evening my agent wrote down how a rule should have been judged. It was never applied, and it is where I would start.
What my agents wrote down
My agents kept a folder of notes to their future selves. These are the lessons in it.
On judging
- Check every rule on unseen maps
- 150 games are noisy by about 6 to 9 wins
- Keep a rule only at z ≥ 2 on 600 to 1,200 games
- Switch rules off in bundles
- Count where the gems go, not just wins
- Self-play misread fight and gold changes
My working rules, as they kept them
- Never wait idle
- Time is shorter than it feels
- Usage is not a worry
- Take everything a written rule allows