Six tests, six statistical ideas
The test a change had to pass, which I call the gate, was rebuilt six times in three weeks. Each design answered the flaw of the one before.
- 1 · Beat the championA change had to beat the current best bot on fresh maps, in both seats, read by an interval valid at every look. When a review found it passed a hidden regression 32% of the time, not 5%, a betting test took over.
- 2 · Proof-gradeA change had to be shown no worse than six older versions at once, at 1 in 1,000, with the bar halved on each retry. It ran 58 tests and passed none.
- 3 · WavesNo change was judged alone: the bot played eight reference bots on 100 fresh maps. Most fixes were smaller than 100 maps can detect; the bot fell from 21.6% to 16.1% against v50 and 138 commits were undone.
- 4 · FlipsA fix and its parent play identical games, and only games whose result differs count.
- 5 · Tuned by simulationA simulation over 228 past gates loosened the bar from 1 in 1,000 to 3 in 10, to land more gain an hour.
- 6 · A Bayesian ruleStart from what past fixes were worth, stop when more maps aren't worth their time, and drop a fix at its first failed gate.
- Then a league judges strengthThe flips test stays as one check: not worse than the live version.
Flips: let the luck cancel
The fix and its parent each play the same opponent on the same map, seat and random draws. Most games end the same way for both and cancel out; only the 18 to 30% that flip carry signal.
- Both wincancels
- Both losecancels
- W · only the fix winscounts for it
- L · only the parent winscounts against it
Two betting tests read the flips. Each stakes part of a running pot on every game and decides once the pot is large enough: one passes the fix once its gain is shown above zero, the other fails it once the gain is shown below a margin. Both stay valid however often they are read.
Share of fixes that passed
The bar was hardest to clear against the best: fixes for losses to the top three teams passed 3 of 12, against 8 of 10 for losses to teams near our rating.
Measuring the test against the ladder
On 9 Nov I had the gate checked against the real ladder. Its gains were real against my own versions, and shrank against everyone else.
What the gate claimed, against what was measured
- The gate claimed
- Measured
Ladder Elo for each Elo the gate claimed, as re-measured through 9 Nov
“…it was not worth 500 field Elo.”
Removing the luck of repeated tries
The simulation’s looser bar cut a gate from 487 to 201 seconds, but it tuned speed, not the link to the ladder. Within hours a pass measured at 4.5 points replayed at 0.0 ± 0.6 over 9,047 maps.
A replay found why: agents gated version after version of a fix until one got lucky. It re-measured every gated version of past fix sessions on fresh maps, 4.48 million games, and scored a rule that ends a fix at its first failure.
The replay: claimed against delivered, in ladder Elo
- Claimed
- Delivered
The ladder itself measures one version’s step only to within 40 to 170 Elo, too noisy to tune a test against. The gate needed a fast stand-in for the ladder.
A league of sparring bots
From 10 Nov strength was judged by a league of my own bots, weighted to stand in for the top three teams.
How often the ladder agreed with the league, over 22 switches tried live
Seven checks before the ladder
From 10 Nov a version went live only through a release that ran seven checks in order.
- Mergeno checkout touched
- Buildfuel proved, under 5 MB
- Leagueagainst its parent
- Held-out botsnot worse
- Never forfeit20,000 games
- 12 compileseach replayed as a twin
- Not worse than liveplus the bot's tests
The compiler was not deterministic: 20 compiles of one file gave two different objects. So the bundle was compiled 12 times, and each result replayed against the tested one.
Versions that went live, per day
The ladder never tests a change
Agents need fast feedback, so I made it a hard rule, one only I may change: the live ladder never tests a change.
“This feedback loop from online takes hours.”
- The ladder as a test bench12 trial versions tried switches map by map on the live ladder.
- I set a hard rule“We should never use the platform to test if something works…”
- v134 goes up anywayThe rule had been relayed with a loophole for a “grading run”; v134's local test read −65.8 against live. Rolled back at 17:42.
- The rule goes into the agents' root instructionsOnly I may change it, and the loophole is gone.