Skip to content
Thomas Fairhurst
Chapters

Chapter 06 · Maps

Maps nobody had seen

With only 128 official maps, a small gain could never be told from luck. So I had a generator learn how the official maps are designed, used AI judges to find what gave its maps away, and ran every test on maps nobody had seen.

128official maps by 6 Nov, all there were to learn from
6tells the AI judges spotted, each turned into a statistic
3.2a held-out detector's score against my maps, down from 6.5 in one evening; 2 is chance

Why I needed new maps

A change helps on some maps and hurts on others. With 15 real maps to score on, no gain below 1.22 times that map-to-map spread could ever be settled, however many games were played.

Official maps on file

I set the rule on 2 Nov: “a generated map that hasn’t been used in training data … should count as an unseen map.”

Four generators in three weeks

  1. 1 · Mix the real mapsSwapped settings between real maps, added jitter and kept a quarter as extreme stress maps, checked by its own statistics.
  2. 2 · The same generator, portedReproduced the first byte for byte, fitted to 55 maps. It passed two formal checks and was admitted on 4 Nov.
  3. 3 · Craft: learn the design rulesWritten after agents read the patterns of the official maps. Three rounds of AI judges that afternoon; at 17:45 it replaced the old generator everywhere, fitted on all 128.
  4. 4 · Statistics onlyPorted to C++, then climbed on 123 measures and a held-out detector.
  5. The league takes overThe map-scoring tools were retired: from then on, the sparring league judged the maps the tests used.

Passing checks was not enough

The second generator passed every check it was given. An audit on 6 Nov showed it was still badly off.

The second generator’s checks

CheckResultVerdict
Distance between generated and real maps0.0085, where chance gives 0.088passed
Twins: results on a real map against 4 generated from its settingsa gap of −2.06 points, SE 1.14admitted, 4 Nov
6 Nov audit: a classifier on 67 measures tells fake from realAUC 0.94 to 0.96, against 0.68 to 0.85 for real against realfailed

The same audit: the bot’s win rate against v50, my first climb’s best bot

The bot against v50
6.4%
24.5%
  • On real maps
  • On generated maps

Generated maps made the bot look about four times as good as it was. The generator had been fitted to 55 maps while the ladder already played 128.

AI judges as the adversary

The judges were my idea: give agents a mix of maps and have them “guess which maps are the ones from the official game”. Whatever gave a fake away became a statistic the generator had to match.

Three rounds of AI judges, 6 Nov

Round 1, 13:28Round 2, 15:11Round 3, 16:10
JudgesOpus 5.5, medium effort6 Opus 5.5, high effortOpus 5.5, high effort
Labelled right, real or fake (95% interval)60% (54.5 to 65.5)62.7% (57.2 to 68.1)61.3% (55.8 to 66.8)
Before the roundone classifier checkabout six versions against a split testthe judges' tells turned into measures
What the judges caughteach fake copied one real map's settings and camps, so they paired fakes with their sourcesno single idea, noise textures, camps strewn evenly, stamped filler, altars unrelated to the layout, missing whimsynew tells that the fixes themselves had created
Round 1 is not like the others: its fakes copied real maps, and without the pairs it matched, judges caught fakes 49% of the time. Its GPT half was cancelled before reporting.

How well a fake could be told apart, before and after the tells became statistics

A statistical detector, AUC (0.5 is chance)
0.776
0.687
A model of the judges' scores, AUC
0.631
0.533
  • Before
  • After

The statistics moved; the judges, at 62.7% before and 61.3% after, did not. Fixing tells one by one made new ones, so when I said the fixes were going in at random, the agent switched from patching to selection.

The critic, from 16:46

4builds a map8 from 8 Nov, 20:25
1critic scores eachby how much it looks like a ladder map
1is drawnin proportion to its score, not the best one

The kind of map it missed

On 8 Nov I asked whether the generator was overfitting. It wasn’t. It was missing a whole kind of map: barrier maps, where holes that a dash crosses lie between the spawns and the camp groups.

Barrier maps, and what missing them did to one test

Share of maps with a barrier
14%
38%
Measured gain of a fix to dashing
+0.10
≈ +0.30
  • My test maps
  • The ladder's real mix

The checks missed it because they scored a hand-picked list of 123 measures one at a time, averaged them so one miss weighed 1/123, and used the maps the generator was fitted on.

“This category of issues should not even be possible to exist.”
8 Nov

A detector that never saw the maps

The answer that evening was general: a detector that tries to tell generated maps from official maps their generator was never fitted on. Its score became the number to beat.

8folds of the 128 mapseach one held out in turn
64maps madeby a generator refitted on the other 7
200label shufflesshow what chance looks like
≈2score at chanceplanted defects read 3.3 to 4.9

The detector’s score, 8 Nov evening (lower is harder to tell apart)

Start
6.54
A statistical critic, wall windows, camps in bushes4.41 on a sealed seed
4.35
The critic chooses among 8 builds, not 42.47 on a sealed seed
3.43
Islands± 0.15 over 32 seeds; 3.01 on a sealed seed
3.19
Chancewhat the label shuffles read
≈ 2
The detector could still tell the maps apart, at 3.0 to 3.2 where chance is about 2. A sealed seed is one kept back from the climb. I stopped at 23:32: “ok you can stop there regarding quality”.

A fresh map for every game

Craft, the design-rule generator, first spent about 8 seconds a map, some 300 times the old generator. A C++ port landed at midnight and made the same map in 0.147 CPU-seconds against 7.93.

Worker time to make one map in C++, in ms

When the speed climb began7 Nov
348 ms
After it7 Nov, 7 landings, the same maps byte for byte
103 ms
When the evening's quality climb began8 Nov
98 ms
Final, the critic choosing among 8 builds8 Nov, 23:45
224 ms

By the end every test game ran on a freshly generated map, a new seed each time, and release checks drew their own. On the last day they caught a branch fitted to ten lost games.

The last day’s branch, 13 Nov: wins in 150 games on 75 fresh maps

Wins out of 150
78
22
  • Before
  • After

Its ten chosen games got better; fresh maps showed what that cost. Judged on them from then on, the branch ended level with v139.