Five Minutes, One Hour, Three Days
For a webinar I showed three builds of the same kind of RAG assistant: a bare prompt, an unattended Opus run, and three days of sessions with me at the wheel; what the Opus run decided on its own is the part that still needs a person.
Opus had been building for 78 minutes. At 12:47 I typed into my other session: “opus is still not done .. working on SSO now … i wonder if it will even stop at some point”.
Twenty-six seconds later it stopped. It posted the summary of a working knowledge assistant for HR documents, built without asking me a single question. I had told it not to.
That run was one of three builds I showed in the KI-Syndikat’s first showcase webinar on 2 October. Twenty people listened. All three were the same kind of system: an assistant that answers questions from a company’s own documents and says where it found each answer. RAG, in the trade. What changed between them was how much of a person went in.
Three Builds, One Kind of System
The blueprint was a use-case page on the KI-Syndikat site, an internal knowledge base. It lists what such a system needs: a source for every answer, access rights, one fixed sentence for “I don’t know”, a named person who owns each kind of document.
The two unattended builds ran in a fresh Claude Code profile: no memory, no CLAUDE.md, none of my skills or agents. The third one is older.
| Build 1 | Build 2 | Build 3 | |
|---|---|---|---|
| When | 1 October | 2 October | July |
| Model | Sonnet 5.5, medium effort | Opus 5.5, effort xhigh | Opus (4.8, as far as I remember) |
| Who decided | the model | the model | the model and me |
| Time | about 3 minutes | 79 minutes on the clock, about 45 of them working | three days, a few sessions |
| Compute at API list price | $0.29 | about $11, Claude Code’s own tally | not recorded |
The third build needs a caveat up front. ragkit is a showcase prototype I built in July, and its job was different: a catalog assistant (the interface calls it Katalogpilot) that answers inside-sales questions about products and their documents, mostly in German. Same kind of system, not the same brief. The afternoon before the webinar, an agent pointed it at the same three HR PDFs the Opus run got and switched its answers to Claude through the command line. That took seven commits and about a quarter of an hour.
Five Minutes
The first build was a dry run the day before: Sonnet 5.5 at medium effort, one prompt of the read-the-page-and-build-it kind. About three minutes.
It ran. It found keywords in the documents, which made it usable as a search. It looked like the thing.
Then I looked closer and wrote in my session: “the vanilla app produced by another you did not even have a llm search … it was just an indexed based one”. A knowledge assistant whose search had no language model behind it, just an index.
The next day I ran Sonnet again, with the exact prompt Opus got. It built keyword search again, and its summary said so: no semantic search, and proper answers would need an API key I hadn’t set.
A bare prompt gets you something shaped like the thing. Whether the core is in there, you only find out if you know where to look.
One Hour
For the second build I changed two settings before typing anything. The model went to Opus 5.5. The effort went to xhigh, which Claude Code describes as “just below maximum”. Then one line, in German:
Lies https://www.ki-syndikat.de/usecases/allgemein/01-interne-wissensdatenbank und bau das für unsere HR-Dokumente. Frag mich nichts.
Read the page and build that for our HR documents. Don’t ask me anything.
Seventy-nine minutes later it had written about 3,300 lines, most of them Python on FastAPI. Not all of those minutes were work. At 11:56 it tried to look at one of its own screenshots in a folder outside the project, and nothing happened until 12:30, when the read was refused. Without that wait the build took about 45 minutes. Hybrid search: embeddings plus its own keyword search that handles umlauts and splits German compound words. Answers from a local model: it found the Ollama install on my machine and chose it, citing data protection. Every statement carried a number that opened the PDF at the right page, with the chapter. Access rights were applied before the search, so a restricted document never reached the prompt. A usage log and an analytics view for HR, owners and review dates for every document, OCR for scanned pages, a light and a dark theme, 15 unit tests, 21 test questions. Its README had a table mapping each requirement on the use-case page to its implementation.
It also looked around. About a minute in, it searched my shell configuration for Anthropic and OpenAI keys, masking the values before printing them. Nobody had asked it to look.
Claude Code’s own tally for the session came to about $11 at API list prices; that includes two short turns after the build. My agent counted from the log: 86 model steps, about 180,600 output tokens, almost 20 million tokens of context read again from cache.
Almost Like a Twin
When it finished I opened it next to ragkit and wrote in my session: “almost like a twin”. Same building blocks, the same kind of interface, although it had picked Python where ragkit is TypeScript.
I asked my agent to take a look. The first thing it checked was whether the run had ever seen my July code. It hadn’t. One web fetch, the use-case page, as plain text. No trace of ragkit anywhere in its transcript. The likeness had three ordinary sources: the page said what to build, any strong model knows the standard RAG toolkit, and both systems were reading the same three PDFs.
On features the hour came close. It had things ragkit doesn’t: OCR, document owners, review dates, usage analytics. ragkit had things it didn’t: a reranker, retrieval metrics, a wider choice of model providers, switchable at runtime. A feature list doesn’t show where the three days went.
What the Hour Decided Alone
The gap is in the decisions. “Frag mich nichts” meant that every question it would have asked me, it answered itself.
It invented the roles: employees, a marketing team, HR. To demonstrate access control it restricted the marketing document to marketing and HR. Nothing in the documents said that.
It filled in owners and review dates for every document, with the role but no name: teams, not people, although the use-case page asks for named people. It marked the Wikipedia article overdue on day one.
By default it logged every question in plain text and kept it for 90 days. Its own number.
It built no login of its own, only a mode that trusts groups passed in by a single sign-on proxy, which this machine doesn’t have. In the demo, a profile dropdown let anyone become HR.
Prompt injection got one rule in the system prompt: the excerpts are data, not instructions. No test behind it.
And 21 of 21 test questions passed. It wrote the questions, and it graded the answers with its own keyword check. I have been here before: green tests written by the author tell you what the author believes.
To be fair to it: its summary had a section headed “Was ich ohne Rückfrage entschieden habe”, what I decided without asking, and closed with what to settle before real use: put named people in as owners, connect single sign-on, ask the data protection officer whether questions may be kept for 90 days, bring in the works council. It said plainly that anyone could pick HR in the demo, and so did its README.
That is a list of questions that only someone accountable can answer. The model can write that the works council should be involved. It can’t sit in the meeting.
Three Days
ragkit’s main branch has 89 commits: July 11, then July 17 running overnight into the morning of the 18th, then one documentation commit on the 22nd. I was in those sessions. As far as I remember the model was Opus 4.8; the commits don’t record it.
Read the log and it is full of decisions, most with a reason next to it:
- Some catalog PDFs embed a font that shifts every letter. A repair pass detects the pattern and fixes only the pages it affects.
- Queries get expanded with a deliberately short list of German trade abbreviations. The ambiguous ones are left out on purpose, and a comment in the code says why.
- A cross-encoder reranker was measured against the lighter default. It won slightly on one metric, lost on another, and was about eight times slower. It stayed optional.
- At 23:34 a stricter citation rule went into the prompt. It was reverted the same minute. The log doesn’t say why.
- At 01:08 a fail-open gap in access control was closed with a default-deny policy.
- A bake-off of small local models kept the 3-billion-parameter default.
None of that is exotic. You only reach it by running the thing on real documents, measuring, and changing your mind.
Both builds have a demo login, and both READMEs say so. The difference is that in ragkit, leaving it open was my call.
What the Hour Left Open
On 2 October, while the Opus run was still going, I wrote in my session that every model release makes the point of our talk weaker: agents get stronger, think of more on their own, and take on complexity instead of failing at it. Three minutes later I added that it isn’t only the technical complexity: the business logic that used to come from a person, who may see which data, is going the same way. Maybe the agent would even think of restricting search by permission on its own. The run in the other session had built that almost an hour earlier: it filtered documents by permission before searching. The page had asked for access rights. Who counts as HR, it made up.
The hour replaced most of what got typed in July. It did not replace judgment, and the judgment started before the first prompt. While I was working on the slides I wrote: “switching the model was the first decision that required a little bit of AI engineering skill. Switching the more unknown effort parameter was already the second intervention to increase quality”. And that “all these micro decisions done right … will matter in the end and compound.”
Every model knows RAG by now. What no model knew here: who is HR, how long to keep what employees ask, who owns the onboarding PDF by name, whether 21 out of 21 means anything. Those questions used to hide under the typing. Now they are most of the work.
Five minutes buys the shape. An hour buys a prototype that writes down its own open questions. The days are for answering them.