Agent ops

AI-orchestrated engineering, measured and reviewed

Most of my recent commits are co-authored by a coding agent, Claude Code, working to specs and plans that I write and review. I say so on every page where it applies. What makes that responsible is the engineering around it: choosing which model does which job by measuring it, keeping an agent inside bounds that code enforces, and keeping the evidence where I can read it again.

This page is that practice. The repositories are private, so it is a written case study with the numbers and the reasoning, and no link to the code. It has four parts: an experiment on which model and effort a task needs, the routing that came out of it and the setup that carries it between my machines, a hook that stops a secret before a model sees it, and a runner that works overnight inside a jail.

A 60-run experiment: does the biggest model earn its cost?

I work inside a subscription plan, so the budget is its usage limits, not a bill. The question was narrow: on everyday jobs, does the largest model at xhigh effort buy anything the cheaper settings do not?

The experiment ran five settings, Sonnet at medium and high and Opus at medium, high and xhigh, over four small task shapes: a bounded code fix with tests, a doc edit, a read-and-answer from the repository, and a short plan under a length cap. Graders scored each run, and a blind judge read the plans. Three runs of each made 60, all usable, and the whole batch cost $13.2 at list price. I ran it twice: once in a clean container, and once on my real setup, with my global instructions, skills and hooks loaded, to see whether the setup changed the verdicts. It did not. Cost below is the list-price equivalent, which I use as a proxy for how much of the plan's limits a run takes. How the plan weighs the models is not documented.

The shape of the result matters more than any one cell. On bounded work Sonnet at high effort matched the most expensive setting for about a third of the cost or less: 0.28 on my setup, 0.35 in the clean container. Medium effort was cheaper again, at the price of one failure in twelve on my setup. My routing table now sends bounded work to Sonnet at high effort.

It does not send lane-shaped work there, and the table says why. A lane is a spec, a plan, an implementation and a review across several files, and the experiment did not run one, so that row of the table reads "extrapolated, not measured". A later replay of three landed lanes found no configuration that passed all three, Opus alone included, so I do not claim more than the row does.

My default effort was xhigh until I measured what it bought.

What I changed, and what the next week showed

The same day I changed two defaults. Effort went to high, with xhigh on demand. And long sessions now compact at 400k tokens of context instead of near the 1M limit, because every turn re-reads the whole context from cache: over the 30 days before, turns at 400k or more were 57% of what my main sessions cost.

A week later I measured again. On my main machine the share of main-session turns at 400k or more fell from 34% to 2%. The effort setting moved less: xhigh was still on 71% of the turns sampled, down from 79%, and the record gives no reason. One week is one sample of different work, so I read each line as consistent or not consistent with its change, and not as proof.

Sub-agents get the cheapest model that does the job

The same habit applies to the agents a session starts for a side job. My rule is a tier rule: complex work to Opus, medium to Sonnet, simple to Haiku, always the cheapest that does the task without quality loss. The routing table gives the measured reason beside each row. Two of its rows have a card.

The first started as plumbing. Claude Code's built-in search agent runs on the main session's model unless the call names one, so on an Opus session every search was an Opus spawn. Whether it had to be was a question I could measure.

The second is the opposite case. A reviewer of a spec or a plan checks a document's claims against the repository, and I wanted to know whether Haiku could do that for a fraction of Sonnet's price.

So the table is not "everything cheaper". Searches go down to Haiku or to a grep, spec review stays on Sonnet, and open-ended design stays on Opus because no cheaper setting has been shown sufficient there. Code review is its own agent, on Sonnet. Write work goes to a small agent that has only the tools it needs, never to one that inherits every tool: that spawn starts with a much larger prompt, 28.3k tokens in my measurement against 6.5k for a short allowlist.

The setup that travels between machines

The routing, the defaults and the rules live in a repository I call claude-global-context. It is the portable part of how I use Claude Code, kept under git so that one setup follows me from machine to machine and I do not rebuild it by hand. It holds:

• Rules that every session reads. They are short on purpose, since every byte of them ships in every session's context. They say how work is routed to agents, how an agent behaves at a fork (take the recommended option, log the decision and go on), and when it must stop and ask me: anything destructive with no copy, a secret, a change to its own permissions, or anything that reaches other people.
• Agent routing by measured evidence: the table this page has been quoting, with the experiment's numbers beside each row.
• Skills, commands and agents for recurring jobs: the reviewers above, and an agent for Unity work that has the engine's rules and drives the editor over MCP.
• Hooks, like the one below. Installing one is a menu choice and never automatic, because a hook runs in every session. The installer touches only entries it recognises as its own, and backs the settings file up first.
• Docs for the decisions behind all of it, with the evidence next to them.

Settings, secrets and anything specific to one machine stay local, and are not in it.

A new machine gets this setup's rules from my install kits for NixOS and Ubuntu, kept in a second private repository, power-scripts. Each kit runs in numbered stages after a stock install. One of them writes the small local file that imports the rules, which no clone can carry, and wires it into Claude Code's own instructions, so every session on that machine reads them. The NixOS kit began as a custom install image instead. Its first install on real hardware took three attempts, and both failures had one cause: the system configuration pointed at copies of my repositories, claude-global-context among them, that existed only on the machine that built the image. The field notes from that install are the main reason the NixOS kit now starts from the stock installer and does its work on the new machine, the route the Ubuntu kit later reused.

Hooks that stop a secret before a model sees it

One hook runs on every prompt I submit, before Claude sees it. If the prompt looks like it holds a password, a key or a token, the hook blocks it: no request is made, the text never enters the conversation, and so no reply, summary or commit can repeat it. The message shown instead says how to hand the secret over by name. The value is typed into a terminal with echo off, and the agent is told only the variable's name.

It looks for vendor token formats, JSON web tokens, bearer and basic-auth headers, private-key blocks with key material after the header, URLs with a password in them, assignments such as password= or the Romanian parola: followed by something shaped like a credential, a password-shaped word near one of those keywords, and a prompt that is nothing but one opaque token. It is built to let ordinary prompts through: Romanian and English text, paths, hashes, commit ids, and code or JSON that merely mentions a password. It deliberately misses a few shapes to keep false positives low, such as an all-lowercase password or a hex-only key. Run over 2,083 of my real prompts, it flagged the 8 real secrets among them and one message that quoted test fixtures. Over 505 held-out texts it flagged none.

Its test suite has 210 checks. Every kind of secret has to fire on a fixture built at run time, so no real-looking token is ever committed. 104 ordinary prompts and pastes have to pass. A false positive can be let through with a marker. The block message never echoes the value. Malformed input has to pass through, and 14 hostile 1 MB pastes must each finish in under 2 s.

Two limits are written in the hook's own documentation. First, it does not keep the text off the machine: Claude Code still records a blocked prompt in its local session transcript. So the block message starts with a fixed marker that a transcript scrubber can find, and a nightly scrubber of mine redacts secrets in my session transcripts. Second, it fails open. An error inside it lets the prompt through, because a checker that broke and blocked every prompt would lock me out of my own session, and a missing script shows a warning on every prompt instead of passing silently. The guard in the runner below makes the opposite choice, for the opposite reason.

The overnight runner

What runs without anyone typing lives in another repository of mine, ai-automations, which has pages of its own. Its overnight runner is the part that lets an agent change code. Each agent job runs a Haiku session in a lane of its own, inside a jail made with bubblewrap, and a guard checks every shell command and every file edit. The guard fails closed, so any error inside it blocks the call. Outside the jail, the runner runs the check itself, in a sandbox of its own, asks Sonnet to review the diff, and pushes only that branch.

What this does not show

It is one person's setup, on one plan. The experiment ran four task shapes at three runs each, near the ceiling for most settings, so it shows the cheaper settings sufficient for those shapes and does not rank the models on hard work. The search and spec-review tests first ran in a cloud container, graded by script or by pattern, and the weekly numbers are a single week. Where a row of the routing table is extrapolated, it says so.

The 60 runs are the batch of 2026-10-18, and the 210 checks are the hook's own suite, which I re-ran on 2026-10-27. The repositories keep growing.