For a while, the simplest prompts in my hobby and learning project were also the most productive: “Yes.” “Continue.” “Do that.” The agent worked its way through my benchmark for local LLMs, and the repository kept getting better.
I always knew what was happening in broad strokes: another task, another phase, another result. But over time, I no longer really knew why we had rejected a plausible alternative, which option had actually failed, or which finding had changed our interpretation.
The project accumulated more and more insights into local LLMs and how to build good benchmarks and evals. But I learned less and less while building it, because I had fallen into the productivity-prompt-just-keep-going trap [© storz ;].
Workflow is not the same as a learning path
The workflow itself was good. The project uses predeclared experiment plans, hashes for exact system identities (in other words: What are we actually testing?), automated tests, deterministic verification, and so on. All good.
But the workflow “AI proposes, I say: Yes, great, do it” quickly stops teaching me anything. I do not have to reconstruct the LLM’s reasoning. This is called cognitive surrender: I delegate not only the work, but also the thinking.
The work itself was solid, but I became passive.
Learning matters more than moving fast—at least here
One phase, for example, exposed two sandbox failures before the test ever reached a real model. It would have been easy to record them as failed benchmark runs. But they were not. The actual process tree differed from our static assumptions, and every relevant change to the harness or its runtime environment creates a new version of the complete system. One of the most important insights: You cannot “simply test an LLM.” The entire infrastructure around the task and the LLM is part of the system being tested.
So: put learning back into the workflow
The first idea was a private learning case study. It reconstructs several important phases of the project through their challenges, alternatives, decisions, failures, and limits of interpretation. In other words, I ask the agent to explain why we did what we did, when we did it—and what we did not do. I should have been doing this all along; now I am catching up.
The second change: a small checkpoint before and after any work that changes the project’s conceptual state:
| Before the work | After the work |
|---|---|
| Question: What are we trying to find out? | Observation: What happened? |
| Scope: Which component or claim are we examining? | Classification: Where in the system did it happen? |
| Options: What materially different alternatives exist? | Surprise: Which assumption changed? |
| Decision: What will we do—and why? | Evidence: What is now supported—and what is still unsupported? |
| Prediction: What result would change our interpretation? | Next step: What is the smallest useful next question? |
The format is deliberately lightweight. It adds friction, and you can ruin a hobby project by making it too complicated: At some point it becomes a chore, and you lose interest. Ahem—at least I do. Gnōthi seauton!
A prediction changes how I read the answer
For me, an important step in learning is the prediction question. If I read the LLM’s explanation first, almost any result sounds plausible and self-evident—partly because I have already surrendered the thinking. But if I have to commit to an expectation first, I have something against which I can compare the observation. The LLM is then not only an implementer, but also a coach.
In this project, questions like these help me:
What poses a greater threat to the validity of the benchmark: a model that is sometimes wrong—or a harness that alters a correct answer during transport?
If three identical test runs produce the same wrong result, will a fourth repetition teach us more than a test built around a structurally different task?
If I had to explain tomorrow, without the chat history, why we chose this path—could I?
This lets me disagree, ask more focused questions, be surprised, and change my assumptions for a specific reason. Much better than simply being pleased with the result.
Y’all, Vibe Coders: The project output alone is not enough!
I still delegate deeply. The agent can carry the implementation state forward across many steps, run extensive checks, and execute a frozen plan without asking for my approval for every command. Giving that up would solve the wrong problem; I do not want to approve more steps at a finer level—I want to learn.
So I now expect two outputs: the project artifact and a compact explanation that advances my mental model, builds my knowledge, and adds to my experience.
I had merely been approving the work while the project moved forward and my understanding fell behind. That is why I changed the collaboration: with a private learning narrative, small learning checkpoints, and prediction questions.
And—painful but true—this applies not only to learning and hobby projects, but also to AI-assisted work in client projects. Review means more than checking the AI’s output. Review means learning what happened and why.