Why I needed new maps
A change helps on some maps and hurts on others. With 15 real maps to score on, no gain below 1.22 times that map-to-map spread could ever be settled, however many games were played.
Official maps on file
I set the rule on 2 Nov: “a generated map that hasn’t been used in training data … should count as an unseen map.”
Four generators in three weeks
- 1 · Mix the real mapsSwapped settings between real maps, added jitter and kept a quarter as extreme stress maps, checked by its own statistics.
- 2 · The same generator, portedReproduced the first byte for byte, fitted to 55 maps. It passed two formal checks and was admitted on 4 Nov.
- 3 · Craft: learn the design rulesWritten after agents read the patterns of the official maps. Three rounds of AI judges that afternoon; at 17:45 it replaced the old generator everywhere, fitted on all 128.
- 4 · Statistics onlyPorted to C++, then climbed on 123 measures and a held-out detector.
- The league takes overThe map-scoring tools were retired: from then on, the sparring league judged the maps the tests used.
Passing checks was not enough
The second generator passed every check it was given. An audit on 6 Nov showed it was still badly off.
The second generator’s checks
| Check | Result | Verdict |
|---|---|---|
| Distance between generated and real maps | 0.0085, where chance gives 0.088 | passed |
| Twins: results on a real map against 4 generated from its settings | a gap of −2.06 points, SE 1.14 | admitted, 4 Nov |
| 6 Nov audit: a classifier on 67 measures tells fake from real | AUC 0.94 to 0.96, against 0.68 to 0.85 for real against real | failed |
The same audit: the bot’s win rate against v50, my first climb’s best bot
- On real maps
- On generated maps
Generated maps made the bot look about four times as good as it was. The generator had been fitted to 55 maps while the ladder already played 128.
AI judges as the adversary
The judges were my idea: give agents a mix of maps and have them “guess which maps are the ones from the official game”. Whatever gave a fake away became a statistic the generator had to match.
Three rounds of AI judges, 6 Nov
| Round 1, 13:28 | Round 2, 15:11 | Round 3, 16:10 | |
|---|---|---|---|
| Judges | Opus 5.5, medium effort | 6 Opus 5.5, high effort | Opus 5.5, high effort |
| Labelled right, real or fake (95% interval) | 60% (54.5 to 65.5) | 62.7% (57.2 to 68.1) | 61.3% (55.8 to 66.8) |
| Before the round | one classifier check | about six versions against a split test | the judges' tells turned into measures |
| What the judges caught | each fake copied one real map's settings and camps, so they paired fakes with their sources | no single idea, noise textures, camps strewn evenly, stamped filler, altars unrelated to the layout, missing whimsy | new tells that the fixes themselves had created |
How well a fake could be told apart, before and after the tells became statistics
- Before
- After
The statistics moved; the judges, at 62.7% before and 61.3% after, did not. Fixing tells one by one made new ones, so when I said the fixes were going in at random, the agent switched from patching to selection.
The critic, from 16:46
The kind of map it missed
On 8 Nov I asked whether the generator was overfitting. It wasn’t. It was missing a whole kind of map: barrier maps, where holes that a dash crosses lie between the spawns and the camp groups.
Barrier maps, and what missing them did to one test
- My test maps
- The ladder's real mix
The checks missed it because they scored a hand-picked list of 123 measures one at a time, averaged them so one miss weighed 1/123, and used the maps the generator was fitted on.
“This category of issues should not even be possible to exist.”
A detector that never saw the maps
The answer that evening was general: a detector that tries to tell generated maps from official maps their generator was never fitted on. Its score became the number to beat.
The detector’s score, 8 Nov evening (lower is harder to tell apart)
A fresh map for every game
Craft, the design-rule generator, first spent about 8 seconds a map, some 300 times the old generator. A C++ port landed at midnight and made the same map in 0.147 CPU-seconds against 7.93.
Worker time to make one map in C++, in ms
By the end every test game ran on a freshly generated map, a new seed each time, and release checks drew their own. On the last day they caught a branch fitted to ten lost games.
The last day’s branch, 13 Nov: wins in 150 games on 75 fresh maps
- Before
- After
Its ten chosen games got better; fresh maps showed what that cost. Judged on them from then on, the branch ended level with v139.