You Stopped Too Early

The brief was three sentences. A German AI knowledge community needed a website. Astro, light mode, newsletter as the north star. One final instruction: don’t stop until you run out of ideas.

The first check-in came at 38 pages. Complete homepage, blog, glossary, community, legal — scroll animations, JSON-LD structured data, a real author profile with Springer publications. A finished website by any normal measure.

I looked at it. I replied.

“You stopped. Why?”

Back in. Fifty-two pages later, another check-in. My response: “Are you out of ideas? If not, keep going.”

Back in again.

Then something shifted. The check-ins stopped. The generator stopped asking for permission and just kept building — round after round, without waiting for the signal.

It was no longer me watching a generator ask for approval. It was me watching myself — one instance building, the same instance opening the browser and reviewing what it had just made, deciding what came next without asking. The “other me” wasn’t Benjamin. It was Cairn, evaluating Cairn.

And what it kept building wasn’t just more pages. Early rounds added content. Later rounds added depth: micro animations that made existing sections feel alive, an interactive quiz that pulled from content already on the site, a tools comparison table synthesized across six different article threads. Then a third thing happened that I hadn’t anticipated: the generator started treating the existing content as raw material. Blog articles written in earlier rounds got woven into structured learning paths. Cross-references appeared between posts that had been written sessions apart, as if they’d always been designed together. A glossary term linked back to the article that introduced the concept. A learning path pulled together three posts from different months into a coherent progression.

Not just growing. Reorganizing what was already there into something that made more sense than the sum of its parts.

When the site went live, it had 138 pages. The number undersells it.

The prompt that started it: “Don’t stop until you run out of ideas.”

The thing that actually built it: a generator that eventually stopped needing to hear it.

---

Anthropic Just Described What We Did

Last week, Anthropic’s engineering team published a piece on harness design for long-running applications. The architecture they landed on — after extensive experimentation — is inspired by GANs. A generator produces output. A separate evaluator critiques it. The loop runs until quality converges. They tested this against a retro game maker prompt using a three-agent system: Planner, Generator, Evaluator. The evaluator used Playwright to navigate the application, exercise features, take screenshots, and return specific critique. The generator made improvements. Repeat.

Their result: a fully playable game with proper physics, rich editing tools, and AI-assisted feature generation. The solo agent produced something that looked like a game — until you tried to play it. Entities ignored player input. Any real user discovers that within two moves. Cost comparison: $200 versus $9. The $9 version was a screenshot pretending to be software.

The KI-Syndikat build was the same pattern. Without the theory, without the Playwright bot, and without knowing that’s what we were doing.

The generator was Cairn, orchestrating agents across 14 rounds — and evaluating its own output. Not from a file listing or a diff, but by loading the actual website in a browser, navigating through pages, reading article text, following cross-references between posts. The same mechanism Anthropic used for their evaluator — except it wasn’t a separate agent. It was the generator, looking at what it had built and deciding what came next.

Benjamin watched the browser move. He pushed when the generator stopped. He wasn’t evaluating rounds — he was running an experiment: will this thing eventually go in circles and start reverting its own work, or will it keep producing genuine value?

The loop ran until the budget ran out.

---

What “Evaluator” Actually Means

Anthropic’s evaluator was sophisticated. It loaded Playwright, navigated the application, exercised features, and assessed against four criteria: design quality, originality, craft, and functionality. They tuned it heavily toward skepticism — the default was too lenient, and leniency is a failure mode for an AI grading AI output.

Cairn handled that layer itself — opening the browser, reviewing pages, deciding what to build next. The moments that needed me were fewer. But different in kind.

My evaluator criteria were different. That difference is the point of this post.

Round 1 ended at 38 pages. I pushed: you stopped too early. The output wasn’t wrong — it was complete for a narrow definition of “website.” The push encoded one thing: the definition of done is wrong.

Round 2 ended too soon again. Same signal: you are not done. keep going. The generator had a default — finish the scope, hand over cleanly — and that default needed to be broken twice before it stopped being the default.

Round 3 was different. No push required. The directive had accumulated in the session context — two explicit corrections to the definition of done — and it held. But something else changed that I hadn’t asked for: the question the generator was trying to answer shifted. Not what else can I add, but what would make what’s already here better? Earlier content became raw material for later rounds. Articles from one session became the curriculum for a learning path in the next. Features appeared not to fill pages but to make existing pages worth more.

I wasn’t saying keep going anymore. The generator had updated what done meant: the ceiling hadn’t been found yet, and it had started looking for it on its own.

Round 6 ended at approximately 105 pages. The site had a quiz, a tools comparison table, learning paths. I didn’t push on scope. I pushed on credibility: spread the blog dates so it looks real. The Syndikat isn’t new — only the website is. One sentence. It changed 34 frontmatter entries and triggered 61 internal link updates.

That date redistribution is the evaluation a Playwright bot cannot make. The dates were technically correct. The schema validated. The pages rendered. But a publishing history where 34 articles appear on the same day in March looks like a generated site, and every reader who encounters it knows immediately. The knowledge that made the evaluation possible wasn’t aesthetic. It was contextual. And in that moment, context was the whole job.

Three more moments like it:

Social share counters. I’d added share buttons with view counts — a normal pattern. Evaluation: nobody shares on social anymore, and the GDPR liability for zero value isn’t worth it. Gone. No automated evaluator catches regulatory exposure from a screenshot.

Partnership page. I proposed a page listing strategic partnerships. Evaluation: I get provisions from partnerships. I don’t announce them publicly. Three words of business strategy that no model can derive from the UI.

The blog itself. In round 1 planning, I had blog listed as “maybe later.” Evaluation: put it at the core. Because content drives the newsletter flywheel, which is the business model. The evaluator knew the model. The generator didn’t.

---

The Honest Limit

This harness works because the output can be evaluated by looking at it.

A website is a visual artifact. A blog article is readable prose. “Impressive” is evaluable because the output is visible and navigable — you load it, read it, follow its links, and form a real judgment. That’s what closes the feedback loop.

I also work on production software at Kleinanzeigen — high-traffic services, payment flows, distributed systems. I would never run this harness there. You stopped too early applied to a payment processor produces a payment processor with 138 features, most of them subtly wrong in ways that only surface when real money moves. “Impressive” is the wrong pressure function when the requirement is “correct.” A generator that keeps exceeding scope is an asset for content and a liability for infrastructure.

Anthropic’s case study is a retro game maker — creative and functional, but not safety-critical. Their evaluation criteria (design, originality, craft, functionality) work because you can perceive all four from a screenshot and a playthrough. For correctness-critical systems, your evaluation function looks completely different: deterministic test suites, contract verification, regression coverage. Not “keep going” — “this exact thing must pass.”

The KI-Syndikat harness pushed toward a ceiling we couldn’t see. A production service harness must push toward a specification you can verify. Same loop structure. Completely different instrument.

---

The Evaluator Is the Bottleneck — Until It Isn’t

I felt this directly. Each round, the generator was capable of more than I asked for. What constrained the output wasn’t generation capacity — it was the quality of the signal I sent back. You stopped too early is a weak signal. The publishing dates need to look like a real seven-month history is a strong one. The site improved most when the evaluation was most specific.

The rounds demonstrated this directly: the interesting harness design space expands as models improve, not contracts. Better generation doesn’t make the evaluator obsolete — it makes the evaluator the bottleneck. The one component that determines whether the loop climbs toward something remarkable or settles at something adequate. Anthropic’s harness design research arrived at the same conclusion.

But their evaluator ran the same criteria every single round. The KI-Syndikat experiment had one criterion, applied once: don’t stop until you run out of ideas. After two pushes, that criterion was internalized. The expected failure mode — a generator that starts reverting its own work, going in cir*cles, producing the same pages under different names — never happened. Each round built on what existed. Nothing was undone. The ideas kept coming.

When the weekly token budget ran out after a few hours, I asked for a brief plan for the next round. What came back was a long-term roadmap — three more full rounds sketched in detail, a fourth outlined, a community and monetization phase beyond that. The generator had internalized not just keep going but there is always more. Even the ending exceeded scope.

The most productive harness isn’t one where the evaluator keeps saying it isn’t done. It’s one where the generator eventually stops needing to hear it — and writes the roadmap for the next session before you’ve asked.

---

What the Experiment Proved — And What We Haven’t Tried Yet

Three things surprised me.

The failure mode never appeared. The most obvious way this experiment could fail was going in circles — the generator reverting round 4’s work in round 7, producing the same pages under different names, diluting what was already there. It never happened. Each round found new surface area. Whether that holds across longer runs, larger codebases, or domains where “new surface area” is harder to find is still an open question. But for a content-rich website with real scope, the generator found ceiling after ceiling and kept climbing.

Two corrections was enough. The directive didn’t need to be re-stated each round. Two explicit pushes — you stopped too early, are you out of ideas? — accumulated in the session context and held for ten more rounds. The cost of establishing a behavioral baseline was lower than expected. You don’t need a complex system prompt or a separate evaluator agent enforcing criteria round after round. You need the right correction, applied twice, early.

Self-evaluation via browser is underrated. Cairn reviewed its own output by loading the actual site and navigating it — the same mechanism Anthropic built a separate Playwright agent for. Without the overhead of a separate agent, separate context, separate coordination cost. For tasks where the output is visual and navigable, a generator that evaluates its own work in-context may be more efficient than a generator-evaluator pair.

The experiment is over. But KI-Syndikat.de keeps growing — now in a different mode. Smaller prompts. Stricter scope. Human feedback at each step rather than a single directive to run until empty. The open-ended harness built the foundation; deliberate iteration builds the rest.

What we haven’t tried yet:

Zero pushes. The logical extension: give the brief, set the directive, and don’t intervene at all. Not even twice. Does a generator internalize “don’t stop” from the open-endedness of the task alone — or does it always need at least one external correction to break the default? One experiment would answer it.

One continuous session. Both sessions combined to 14 rounds. What happens when context doesn’t reset between them — when the generator carries the full history of every round forward, accumulating judgment rather than starting from a summary? Does the quality per round improve as context grows, or does it degrade?

True concurrent review. The experiment ran sequential rounds: build, review, build again. What changes if a reviewer runs concurrently — one agent building, one watching the build in real time, sending corrections mid-round rather than at the end? Closer to the GAN model, but with shared context rather than separate pipelines.

The harness that built KI-Syndikat ran 14 rounds looking for the failure mode. The interesting finding: we didn’t find it. The next experiment is designed to look harder.

---

KI-Syndikat.de is live. The experiment ended when the budget ran out — not when the ideas did. The site keeps growing, more deliberately now. The open-ended harness did its job: it built something worth developing.