What Does It Actually Verify?
Agents had pinned the exact output of a seeded game simulation all over its tests, and we deleted those pins in one night; the first release measurement afterwards found a blocker the whole suite had been blind to.
At 21:19 on the 4th of September, an agent committed this:
test(pins): refresh all 42 pins on the composed 0.45.0 tip
It was the fourth commit like it that day. The others had refreshed 47, 24 and 41 numbers. Each number was the expected value of a test, rewritten to whatever the code now produced.
About an hour later I typed this into the session:
but run a 5 minute test to see if the city still has 4534 units of wood after 10 game years … that is a waste of time and the worst kind of test
In the same message came the question this post is named after: “What does it actually verify? That we did not change the sim when we actually changed the sim?”
Within the hour, the agents had deleted 148 pinned numbers from 33 test files. Before the next release went out, one comparison against the previous release found a blocker the whole suite had been blind to.
A World You Can Replay
Lastadie is a browser trading game that agents built for me over 33 days, from early August to early September. I wrote about how it started separately. Ships, thirteen towns, fifteen goods, rival merchant houses, and a simulation that runs the whole economy day by day. The simulation is deterministic and seeded: the same seed gives the same world, every time, down to the last barrel of beer.
That makes testing easy, and it sets a trap. If the world always comes out the same, the cheapest test anyone can write is: run it, read a number, assert the number. That is a pin. Here is a real one, from a test about what a town holds on day zero:
expect(held, 'luebeck opens with a different salt pile — … measured 38,993')
.toBeGreaterThanOrEqual(33_000);
expect(held, 'luebeck opens with a different salt pile').toBeLessThanOrEqual(45_000);
expect(Math.floor(cap), 'the salt cap moved').toBe(47_700);
The band around 38,993 was read off one run. So was the cap. Five days earlier the same test had held 16,180 to 26,966 with a cap of 69,840. Now read the failure messages: “a different salt pile”, “the salt cap moved”. A test like this can tell you exactly one thing: a number moved. And we were tuning the economy every day. Moving numbers was the job.
The Bill
An agent asked to cover a change reaches for the assertion it can check on the spot. In a replayable world, that is the pin. Then the next change to the economy lands, and dozens of those numbers move at once, sometimes hundreds. The agent who makes the change has to look at each one and decide: did my change mean to move this, should the test watch a different number, or is this a real bug? That triage cost around 300,000 tokens, and a run of the simulation tests took seven to eleven minutes. In the four weeks before that night, more than 150 commits in the repository had a subject line about re-pinning, refreshing or re-deriving those numbers.
First, I Made Pinning Cheaper
I had seen the cost five days earlier. My suggestion was to treat pins as observations: a refresh command and a diff table, instead of loops where agents re-derived every number by hand. The agents built it as a pin registry that same day.
The same evening, a change that made ships last longer moved 40 assertions in 23 files, the first batch of moved numbers to go through the registry. It took three agents about a million tokens. The registry gained six entries. The agents had re-derived the rest by hand anyway, despite the workflow in their brief.
The agents’ next plan was a codemod to move every remaining observed number into the registry, so there would be nothing left to edit by hand.
None of this was new. Twelve days earlier the agents had deleted a hash over the whole game state because it “pins the one thing tuning is supposed to move”, and since that day the game’s CLAUDE.md had said: “A permanent pin is almost always a mistake.” The cleanup that followed turned dozens of pins into corridors, bands instead of exact values. The salt test above is one of them. A corridor is a pin with a wider tolerance; it still asserts what one run happened to produce. The rule was written down, and the pins came back anyway. Everyone, me included, was working on keeping the numbers cheap.
The Census Already Knew
The orchestrating agent did not simply accept my ruling. It first checked the rule against the census from five days earlier, which had compared the last 60 re-pinning commits with the bug fixes in the last 400.
Of the 52 test files those re-pinning commits had touched, 32 had never once been paired with a fix. They kept moving, and never caught a bug. And 40 files had caught real bugs without ever moving with the numbers. All of them tested functionality.
So the rule went into the game’s CLAUDE.md, in capitals: no assertion whose expected value was read off a run. Not refreshed, not re-picked, not widened into a band. Deleted. What stays: commands and their refusals, conservation identities, property tests, pure functions, designed caps compared to a constant, UI behaviour, end-to-end flows, battle mechanics (the mechanism, never a rate).
The salt test kept one line:
expect(held, 'the day-0 pile must sit under its own cap').toBeLessThan(cap);
That is the rule the band had been standing in for. The test for the tests had sat in the game’s CLAUDE.md since 23 August: a test earns its place if it can fail for exactly one reason, and that reason is a behaviour someone could notice. That night was the first time the suite was held to it.
One Night
The first simulation commit removed 148 assertions but only a handful of whole tests. The next run turned up 45 more, then measured bands, rates and day-zero counts. About twenty files that mixed a mechanism with a measured number kept the mechanism. This was not “delete the tests”. It was “delete the photographs”.
Ten suites that boot a world and run it for years of game time moved to an on-demand lab, and the registry and its tools went. The simulation tests dropped from 598 seconds to 236 and the UI tests from 149 to 129; by the end of the night they ran in 211 and 109. Part of that came from the lab move and from doubling the runner’s workers, not only from the deletions.
Measure Once, at the Release
Twenty minutes later I sent the second half of the ruling. I did not want the world unmeasured, only measured at the right moment:
i don’t mind something like ONE sim run per seed and a readout (e.g. old balance vs new balance how did the world change?) … but that is a balance round, it has nothing to do with a green CI
Before a release: one run per seed, the candidate against the last release, and a readout of how the world changed. One question, asked once per release and outside CI: did we break the game for the player?
The first balance round ran that same night: three seeds, three game years each, the candidate against 0.44.0, which had shipped that afternoon.
The towns had never looked better. On every seed all thirteen were alive at every checkpoint, fourteen of the fifteen goods were in stock somewhere (the old release had nine to twelve; spices, the fifteenth, was missing on both), and after three years the towns were four to five times richer and more than twice as populous.
The second readout came from a reference bot, a scripted player that buys, sails and sells in a loop. Its ship wore down, as ships do, and when it asked a shipyard for repairs, the game refused for lack of materials. On the first seed that happened 6,716 times; on the old release, 119. On every seed the bot made no trade at all in its third year, in a world holding more than 110,000 units of wood.
The cause: towns protect raw materials their own workshops need, so a trader cannot buy them out. The richer economy tripped that protection everywhere; a town counted as short of timber with 2,310 units on the shelf. The repair code bought its wood and hemp through the same counter as a trader. The fix: the shipyard uses its own store first, then buys from the shelf at the posted price, past the protection.
The existing release gate had stayed green. Its scorecard, a band check on the same bot, read net worth at day 730, found it far above the floor, and passed. It never looked at year three, or at the ship.
0.45.0 shipped with the fix. It cleared two seeds. On the third, the bot’s route still ran through two towns with no hemp at all, and that went to me as a design question.
Three days later, for 0.46.0, a redesign of the screens that left the economy alone, one command compared the candidate’s world with the 0.45.0 tag: byte-identical, and the three-seed scorecard matched line for line. It hashes the whole world after a long run, the same kind of hash the agents had deleted on 23 August. When the world has not moved, one hash answers the balance round. The full round is for a world that has.
The reference point is the whole difference. A pin compares today’s commit with whatever run the last agent happened to see. A release measurement compares what players will get with what they have.
Where Pins Still Belong
In June I wrote approvingly about a golden test that pins a code generator’s output byte for byte. A generator’s output is a contract: when it changes, every user’s code changes, and you want a test to stop you. A game economy in development is what you change on purpose every day.
Pin what is supposed to stand still. Measure what you are trying to move.
What I Am Less Sure Of
A pinned number does fail when something breaks. It also fails for every change that was meant. My nearest line from that night: “they will just fail and need a new calculated meaningless value.” The census found one test file that fired on all three unrelated mutations it tried. That is evidence, not proof: the mutation sweep that could have measured it ran 3 of 20 planned mutations before it was stopped to keep the machine quiet.
The balance round is a reading and gives no pass or fail. It found the stranded ship because the agent running it chose to record ship condition and repair refusals for the bot. Someone else could have looked at the towns, seen the richest world the game had ever had, and shipped.
And the habit has to stay broken. I can’t show that it did: the four days of commits after that night were a redesign of the screens, the economy never moved, and on 8 September I stopped. The rule had been written down once before and had not held. That is why it now sits in capitals: the cheapest assertion will always look like coverage.
Ask this of every test an agent hands you: what would have to happen for this to fail? If the only answer is that someone changed what they meant to change, it is not guarding anything. It is a photograph of the last run.