Skip to content
Thomas Fairhurst
Chapters

Chapter 02 · The machine

A server for the agents, a cluster for the games

My agents ran on a remote server, so closing my laptop stopped nothing. A gateway spread their requests over 7 Claude subscriptions, and their games ran on Imperial's clusters, pulled by workers with no central dispatcher, at up to 1,906 games a second on freshly generated maps.

7Claude subscriptions behind one gateway
4,916threads on Imperial's clusters on 10 Nov, 772 of them running my games
1,906games a second at the peak, 8 Nov

The agents never lived on my laptop

Every agent ran on a remote machine through T3 Code, an app that keeps agent threads on a server. I wrote to them from my laptop or my phone. After two Imperial hosts in four days, they settled on my own server on 26 Oct and stayed there.

The three hosts

Lab machineBatch serverMy server
When22 Oct22 to 26 Oct26 Oct to 13 Nov
Sizenot recorded32 cores, 64 threads8 cores, 16 threads, 62 GB RAM, 467 GB disk
OwnerImperialImperialme, shared with Leadlord, my company
WhereImperialImperialMontreal, 79 ms from Imperial
Why it endeddisk full of system logsslow, degraded shared disksran to the end

The server also played the live platform for me. A queue asked for five-game practice matches every 72 seconds, within the platform’s quota of 100 an hour.

Practice match groups requested from the live platform, per day

One gateway, seven subscriptions

Every agent’s request went to one gateway, shared with Leadlord and reachable only over a private network. It picked a Claude login for each request, so the work drew on all of them at once.

The logins on the gateway, 10 to 12 Nov

  • Serving
  • Not servingone waiting for a new sign-in, one rate-limited or unavailable
The count moved with the day: on 3 Nov an agent counted six. What a week of one login buys is in Models.

Games ran on Imperial’s clusters

Almost every game ran on Imperial’s Department of Computing machines: the lab desktops when nobody was using them, and two clusters’ CPUs. After a first SSH script, games moved to the university’s batch queues on 28 Oct, and from 1 Nov my rule was all of Imperial’s compute, never my laptop.

Threads visible on 10 Nov

Lab desktops (HTCondor)one job takes a whole machine
2,852
GPU cluster, main limitCPUs only, for games; our cap 112
1,368
CPU clusterour cap 288 CPUs
504
GPU cluster, temporary limit
192

Workers pull their own work

No central machine handed out games: the department’s rules ask for that, and it leaves nothing in the middle to fail. Agents drop games into a shared queue of folders on Imperial’s network disk, and cluster jobs, which I call pieces, claim batches from it themselves.

  • An agent queues games
  • A service sizes the pieces every 10 s
  • A piece claims a batch
  • Plays it
  • Files it as done

What the pool does when something fails

WhenWhat the pool does
A piece diesits batches are freed, up to 2 times
A piece is evictedit is cancelled and its batches freed; the same if suspended
A batch stragglesone backup play, as MapReduce does
A task is stoppedit runs again elsewhere, twice at most
A job is warnedit saves a checkpoint and requeues itself
Output-file holdreleased automatically: 12 of 5,982 jobs, 31 Oct to 6 Nov
No games lefta machine is held idle at most 180 s

From 6 Nov agents ran all of it with one command, and never chose a machine, a CPU count or a memory size.

From 80 to 1,906 games a second

Throughput rose more than twentyfold in a week. Every gate and study game was played on a freshly generated map, so the map generator had to keep up too.

  • Aworkers busy making maps, 7.3 s a map against 0.58 s a game, until the generator moved to C++ on 7 Nov
  • Ba silent fallback to the official engine
The 8 Nov peak was 1,803 games a second from the lab desktops and 103 from the GPU cluster, counted from each piece’s own tally. Where a check gave a range, the line shows its top.

Keeping one server usable for everyone

Sixteen threads carried every agent, every build and Leadlord’s own work. Most guards answered a real failure, and I removed the ones that slowed the work more than they protected it.

  1. A memory cap on every commandAfter one agent's 56.5 GB process took the whole app down.
  2. Work refused below 35 GB freeAfter the disk filled once.
  3. Kill one process, not the sessionOur processes die first, and a memory gate admits heavy steps by estimated memory.
  4. Lowest CPU and disk priorityNo swap; agent commands run below everything else.
  5. Hard memory caps40 GiB for all our work, 16 GB for the analysis database, after one query grew to 55.7 GB.
  6. The memory gate deleted40 GB was free, yet steps started one at a time and builds waited 18 minutes.
  7. The 10-second hookAny agent command past 10 s moves to the background, so no agent's turn waits on it.
  8. A valve instead of a CPU budgetEvery 20 ms it keeps our threads off the cores others are using; 14 CPUs when it isn't running.
“Why do we even have Memgate?”
6 Nov
“We shouldn't have commands that slow down everything else.”
11 Nov

No silent fallbacks

The hardest failures to catch raised no alarm. On 9 Nov a step that quietly fell back to the official engine played four gates 7 to 8 times slower. I deleted the slow path and made it a rule: “No silent fallback at all.”

Numbers that looked right and were not

WhatSeen asReallyWhy
4 gates, 9 Novnative enginethe official engine, 7 to 8× slowernative bot files carried the wrong tag
Games/s, 8 Nov7,3411,906the dashboard divided several minutes of batches by 30 s