Skip to content
Thomas Fairhurst
Chapters

Chapter 03 · Speed

Fast feedback came first

An agent improves only as fast as it can test. So I made feedback time the first problem: my agents hill-climbed their own tools against a single number, and every speedup had to give the same results, byte for byte, as the slower original.

~15×less CPU a game than the official runner, same results
54×faster map generator, the same maps byte for byte
~25×faster fuel proof, 64 s down to 2.6 s

A game was mostly start-up

The official runner compiles both bots before every game. On 4 Nov that was almost the whole cost of a game.

  • Start-up and the first round3.04 CPU-s, 3.0 of it compiling the two bots
  • The other 1,169 rounds0.32 CPU-s

Compile once, play many games

The official engine ships as WebAssembly. A translator, wasm2c, turns it into C that compiles to native code; each bot gets the same treatment once, then plays thousands of games. That alone made a game about 9.5× cheaper. My agents’ rework of the engine then took it to about 15×.

CPU-seconds a game

4 Nov, native code, 2,400 games
3.36 s
0.35 s
5 Nov, after the agents' climbs, 3 games run twice
2.49 s
0.16 s
  • Official runner
  • Our engine

How it got there

  1. The official engine, compiled to native code7.17 games a CPU-second against the official runner's 0.77: 9.4× cheaper.
  2. I flag it as the big leverThe plan had sent every game back to the official runner. “Every single test that we will run will take 2.5 times the amount of time… a huge lever.”
  3. Native becomes every caller's default2,400 games at 0.352 against 3.357 CPU-seconds: 9.5×. “We shouldn't even be able to use the old engine.”
  4. A faster-compiling runtime, rejectedWasmtime compiled a bot in about 1.5 CPU-s against clang's 16 to 28 s, but ran turns 1.36× slower. “Even 30% is big”, so native stayed.
  5. Agents hill-climb the engine's own codeTwo climbs cut 21% and 19%. The ratio to the official runner reaches about 15×.
  6. Build flags and one C call a game0.7914 to 0.7177 worker-seconds a game, 9.3% less.
  7. No fallbacksFour gates had been quietly playing on the official runner, 3.4 to 4.7 s a game against 0.4 to 1.0. Now a game our engine can't play exactly fails and says why.

Agents climbed the engine against one number

On 5 Nov I asked for “something to hill climb”. A profile of one native game showed where its 0.167 CPU-seconds went. Everything but the bots’ own code was fair game.

  • Engine answering the bots' calls
  • The bots' own codeoff limits
  • Copying call buffers and import glue
  • Rules, setup, hashing and output

The biggest find: one engine call copied 0.25 to 1.7 GB a game to read 2 bytes. It now reads the bot’s own buffer. I asked for 70%; with the bots’ code untouchable, about 20% a climb was the ceiling.

What each change cut

14 read calls keep their answer for the turn, 6 bitboard kernels on the bot's own boardsinstructions, 5 Nov
−19.1%
Built for x86-64-v2instructions, 5 Nov
−6.5%
A whole game in one C call, not about 7,700 Python callsworker time, 7 Nov
−3.4%
No frame pointers, no PLT callsworker time, 7 Nov
−2.6%
Call depth counted only in the 138 of 916 functions that can recurseworker time, 7 Nov
−2.5%

Tried and dropped

  • ✗ -O3builds identical to -O1
  • ✗ Segue, one-file buildabout 1% and 0.3%
  • ✗ Profile-guided buildno profiling runtime
  • ✗ Shadow casting in C30% slower than wasm2c's
  • ✗ Shadow-casting memofailed the exactness check
  • ✗ Hand-written portnever played a bot; deleted 6 Nov

Not one game changed

A faster engine is worthless if it plays a different game. Every check compared it with the official engine, and every speedup with the version before it.

WhenCheckGamesDiffered
26 OctWhole engine memory hashed every round, against the official module run in V83,498 games, 3.77M rounds0
26 OctReplay and result file, byte for byte, against the official runner269 of 280 (11 excluded by rule)0
29 OctReplays with fuel metering on5,074 of 5,0740
4 NovOutcome, rounds, faults and result file2,013 played natively0
4 NovThe same on 80 generated maps2,352 of 2,4000
4 NovReplay, result and both fuel files, byte for byte1760
5 NovEvery fast path also run in the original engine192 + 1,890 on 135 maps0
8 NovMemory-hash agreement test, full run181 + 7 end scenariospass

Before it landed, each speedup also had to leave every output file of 192 reference games identical.

Three things it could not play exactly: another champion’s vision (drifts after about round 20), OCaml bots, and some host charges never measured. Until 9 Nov those went to the official runner; since then they fail.

Then the same method on every tool

Two slow Python tools were rewritten in C++ and checked against the originals: the map generator made the same maps byte for byte, and the fuel proof gave the same answers on 114 of 115 test programs.

Python against the C++ rewrite, seconds

Making a map from scratch, 6 to 7 Nov
7.93 s
0.147 s
Proving v50's program fits its fuel, 2 Nov
64 s
2.6 s
  • Python
  • C++
The slow generator had workers making maps instead of playing: 36 games a second on about 800 workers on 6 Nov, against 1,906 at the 8 Nov peak.

On 7 Nov I asked for “a sub one second command” that scores a real, fully simulated scenario. Its rules became the rules of every climb.

1numbercounted in instructions, not wall time
=main's outputsbyte for byte, or the try scores INVALID
?hidden test casessealed, scored once before landing
0edits to the evalby the agent climbing it
  • One change
  • Score it
  • Kept only past the noise
  • Every try logged
  • Land at the bar

From 9 Nov a different agent designed each eval from the one that climbed it, and a climb under 10% after an hour restarted in a fresh agent.

“They shouldn't have the limiting beliefs from their past fails.”
9 Nov, on restarting a stalled climb

Every tool climb, before and after

Waiting on the fuel proof, seconds
136.6
44.8
Launch to verdict, simulated seconds
57.05
29.59
Making a map, worker-ms
348.35
102.98
Rebuild after a small change, seconds
13.62
8.84
Playing a game, worker-seconds
0.7914
0.7177
  • Before
  • After
The verdict figure came from a simulator whose own error was 24%. By 10 Nov real jobs ran slower than it predicted.

Every rebuild got cheaper

Every change to the bot meant a rebuild, so builds got the same treatment.

Building the game check's 528 bot modules, CPU-s
~7,400
3.5
A build through the official runner, CPU-s
66.4
6.6
Module test, seconds
15.3
2.3
Recompile after a fuel-guard change, CPU-s
55
16
Full test suite on a rented machine, seconds
220
68
  • Before
  • After

Where climbing stops working

A climb works when the number is the goal. Aimed at the bot’s play, it moved the score and not the games, so I kept it for tools and deleted the rest.

ClimbIts scoreWhat games said
Fuel a turn56,731.8 → 54,259.6+0.23 Elo, inside −1.26 to +1.71
Turns decided by the fuel limit0.7189% → 0.2109%50.7% head to head, 32,000 games
One farming turn against a deep solve3.2879 → 3.2808noise; its gate failed
Sharpness of the rating model2.1005 → 2.0321the live version's interval widened, ±32 to ±39