The agents never lived on my laptop
Every agent ran on a remote machine through T3 Code, an app that keeps agent threads on a server. I wrote to them from my laptop or my phone. After two Imperial hosts in four days, they settled on my own server on 26 Oct and stayed there.
The three hosts
| Lab machine | Batch server | My server | |
|---|---|---|---|
| When | 22 Oct | 22 to 26 Oct | 26 Oct to 13 Nov |
| Size | not recorded | 32 cores, 64 threads | 8 cores, 16 threads, 62 GB RAM, 467 GB disk |
| Owner | Imperial | Imperial | me, shared with Leadlord, my company |
| Where | Imperial | Imperial | Montreal, 79 ms from Imperial |
| Why it ended | disk full of system logs | slow, degraded shared disks | ran to the end |
The server also played the live platform for me. A queue asked for five-game practice matches every 72 seconds, within the platform’s quota of 100 an hour.
Practice match groups requested from the live platform, per day
One gateway, seven subscriptions
Every agent’s request went to one gateway, shared with Leadlord and reachable only over a private network. It picked a Claude login for each request, so the work drew on all of them at once.
The logins on the gateway, 10 to 12 Nov
- Serving
- Not servingone waiting for a new sign-in, one rate-limited or unavailable
Games ran on Imperial’s clusters
Almost every game ran on Imperial’s Department of Computing machines: the lab desktops when nobody was using them, and two clusters’ CPUs. After a first SSH script, games moved to the university’s batch queues on 28 Oct, and from 1 Nov my rule was all of Imperial’s compute, never my laptop.
Threads visible on 10 Nov
Workers pull their own work
No central machine handed out games: the department’s rules ask for that, and it leaves nothing in the middle to fail. Agents drop games into a shared queue of folders on Imperial’s network disk, and cluster jobs, which I call pieces, claim batches from it themselves.
- An agent queues games
- A service sizes the pieces every 10 s
- A piece claims a batch
- Plays it
- Files it as done
What the pool does when something fails
| When | What the pool does |
|---|---|
| A piece dies | its batches are freed, up to 2 times |
| A piece is evicted | it is cancelled and its batches freed; the same if suspended |
| A batch straggles | one backup play, as MapReduce does |
| A task is stopped | it runs again elsewhere, twice at most |
| A job is warned | it saves a checkpoint and requeues itself |
| Output-file hold | released automatically: 12 of 5,982 jobs, 31 Oct to 6 Nov |
| No games left | a machine is held idle at most 180 s |
From 6 Nov agents ran all of it with one command, and never chose a machine, a CPU count or a memory size.
From 80 to 1,906 games a second
Throughput rose more than twentyfold in a week. Every gate and study game was played on a freshly generated map, so the map generator had to keep up too.
- Aworkers busy making maps, 7.3 s a map against 0.58 s a game, until the generator moved to C++ on 7 Nov
- Ba silent fallback to the official engine
Keeping one server usable for everyone
Sixteen threads carried every agent, every build and Leadlord’s own work. Most guards answered a real failure, and I removed the ones that slowed the work more than they protected it.
- A memory cap on every commandAfter one agent's 56.5 GB process took the whole app down.
- Work refused below 35 GB freeAfter the disk filled once.
- Kill one process, not the sessionOur processes die first, and a memory gate admits heavy steps by estimated memory.
- Lowest CPU and disk priorityNo swap; agent commands run below everything else.
- Hard memory caps40 GiB for all our work, 16 GB for the analysis database, after one query grew to 55.7 GB.
- The memory gate deleted40 GB was free, yet steps started one at a time and builds waited 18 minutes.
- The 10-second hookAny agent command past 10 s moves to the background, so no agent's turn waits on it.
- A valve instead of a CPU budgetEvery 20 ms it keeps our threads off the cores others are using; 14 CPUs when it isn't running.
“Why do we even have Memgate?”
“We shouldn't have commands that slow down everything else.”
No silent fallbacks
The hardest failures to catch raised no alarm. On 9 Nov a step that quietly fell back to the official engine played four gates 7 to 8 times slower. I deleted the slow path and made it a rule: “No silent fallback at all.”
Numbers that looked right and were not
| What | Seen as | Really | Why |
|---|---|---|---|
| 4 gates, 9 Nov | native engine | the official engine, 7 to 8× slower | native bot files carried the wrong tag |
| Games/s, 8 Nov | 7,341 | 1,906 | the dashboard divided several minutes of batches by 30 s |