How I Taught My Coding Agent to Question Its Own Work

The version that interviewed me

The first version of the Socratic skill did what its name promised, literally. Before an agent started a task, it asked me questions. What exactly is the goal? What does done look like? Which assumptions am I making? It felt rigorous. For a week it felt like working with a thoughtful colleague.

By the second week I was answering the same questions for the third time, and some of them were questions the agent could have answered itself. The moment that ended it was small. I asked for a fix to a failing test, and the agent asked me which test framework the project used. The config file for it was sitting in the root of the repository, one command away.

That was the flaw in the design, and it was not about politeness. The skill was sending doubt in the wrong direction. It asked the person things the code could answer, and it never asked the code anything.

Doubt that ends in a verdict

The research I built on says something uncomfortable about self-reflection in language models: an agent that only doubts itself, with no external signal, gets no better, and sometimes worse. A model judging its own reasoning shares that reasoning's blind spots. "Double-check your work" mostly produces a longer explanation of the same mistake.

So version 0.3 turned the skill around. It stopped interrogating me and started cross-examining the work, under one rule that everything else hangs on: every self-question has to end in an external verdict. A run, a test, the type checker, a constructed counterexample, the real source. If a doubt cannot be settled by something outside the model's head, it is dropped, not narrated.

Predict, then run

The core move is borrowed from how careful engineers debug. Before the agent runs anything that tests an assumption, it commits to a prediction: this test will fail on line 42 with a null, this query will return three rows, this migration will leave the old column in place. Then it runs it.

If the result matches, the agent carries on, now with a reason for its confidence. If it does not, the gap between the prediction and the result is the finding, and the plan changes before any more code is written on top of a wrong belief. The prediction is what makes the difference. An agent that runs a test without predicting the outcome will explain whatever it sees. An agent that predicted something else has to notice.

The old interview questions did not disappear. They moved to where they belong. Some questions really are the user's to answer: what the product should do, which trade-off the business prefers, whether a risky change is acceptable. The skill now separates those from the ones the repository can answer, checks the second kind itself, and only brings the first kind to a person.

Gated, so it stays out of the way

A skill that questions everything is as tiring as one that questions the user, so the scrutiny is gated by what is at stake:

And it is mostly silent. The cross-examination happens in the agent's reasoning. What you see is the result: a plan that changed because a check disagreed with it, or a question that genuinely needs you. The honest cost is tokens: an agent that thinks a little longer and runs extra checks uses more of them per session, which is why trivial work is left alone.

Scoring what it did, not what it said

Testing a skill like this has a trap in it. A model is very good at writing a paragraph about how carefully it checked something. So the evaluations in the repository do not read the agent's prose at all. They read its tool traces: did it actually run the test, did the prediction come before the run, did a failed check change the plan. A beautifully worded "I verified this" with no run behind it scores zero.

One command for every agent

The first release only knew Claude Code. But the SKILL.md format is read by a growing list of agents, and developers switch tools more often than they switch habits. So the installer now copies the skill into the folders each agent actually reads, and with no flags it covers all of them at once: Claude Code, Codex, Cursor, VS Code with Copilot, Gemini CLI, Windsurf, Amp, OpenCode, Roo Code, Kiro and Factory.

npx socratic-method            # this project, every agent
npx socratic-method --global   # every project on this machine
npx socratic-method --tool cursor

In a plain chat window with no skills folder, the same file works pasted into a project's custom instructions. The code is on GitHub under the MIT license, and the skill's page has every install option.

What it changed in the agents I build

Writing this skill clarified something about all agent work, not just coding. The agents that go wrong in production are rarely stupid. They are confident about something nobody checked. A scheduling agent I built once read a failed calendar lookup as a free day, which I wrote up as empty is not free. A coding agent assumes the test framework. A support agent assumes the order exists. The fix is the same every time: the agent does not believe it until something real confirms it, and the questions that only a person can answer go to a person.

That is how I build agents for businesses too, whether they read an ERP, a helpdesk or a calendar. If that is the kind of agent you need, it is AI agent development, and the wider patterns are in agent design patterns that hold up.

Frequently asked questions