Back to the network

Tooling

ai-automations

What runs while no one is typing: an overnight runner whose agent jobs work in a jail behind a guard that fails closed, 33 jobs, weekly reviews and local-first tools.

A lane of its own, and a check that runs first

An agent job is a task in one of my repositories that an agent may work on while I sleep, most often "this suite is red, find out why and fix it". Each job gets a lane of its own: a fresh git worktree on its own branch, cut from the repository's main branch. The job's precheck runs there first, and a green one ends the job at no cost. An agent job runs only for a repository that had a session of mine on that machine in the last 14 days. No job starts after the night's deadline, 06:30 at my desk, and no agent job starts once the night's agent jobs have spent $10.

An agent job of the overnight runnerAn agent job starts from a queue, when no session of mine is active, and gets a lane of its own: a fresh worktree and branch, with its check run first. If the check is red, a Haiku session works on it inside a jail, where the filesystem is read-only except the lane and a few places the session itself needs. A guard checks every shell command and file edit and denies pushes, deletes, installs, sudo and deploys, and edits outside the lane; any error blocks the call, so it fails closed. Outside the jail, the runner runs the check itself, has Sonnet review the diff, pushes only the lane's branch, and writes a morning brief that lists what needs me first.An agent job, overnightQueue, then a lane of its ownstarts when no session of mine is activea fresh worktree and branch for each jobits check runs first; a green one costs nothingAgent sessiona Haiku session workson the failing check;it writes only in itslane and what thesession itself needsGuardchecks every shellcommand and file edit;denies push, delete,install, sudo, deploy.An error blocks it:it fails closedRunner, outside the jailruns the check itself, sandboxed, not on trustSonnet reviews the diff; only the lane is pushedthe morning brief lists what needs me first
An agent job, top to bottom. The framed box is the jail: the agent writes only in its lane and a few places the session itself needs, and the guard checks every shell command and file edit and fails closed. The runner sits outside it, and checks, reviews and pushes for itself.

The jail

A red check starts a Haiku session inside a jail built with bubblewrap. The whole filesystem is mounted read-only, with a private /tmp. The session can write only in the lane, the repository's git folder, Claude Code's own state, and any path the job file names. The desktop's runtime sockets, an SSH agent and the keyring among them, are hidden from it, and so are the usual stores of SSH and GPG keys and of cloud command-line logins. The session has no web tools and no MCP servers, every plugin is switched off, and it stops at the job's limits of turns and dollars, 80 turns and $3 unless the job file says otherwise.

Inside the writable parts, some files are mounted read-only again. Claude Code's settings, instructions, hooks, agents and skills are among them, and so are the git folder's config, hooks and info, where a line planted by the session could run outside the jail, in a later git command on that repository. The runner also takes a digest of those three before the session and after it. If they changed, nothing is checked or pushed, and the brief tells me to inspect them before I run any git command in that repository.

If bubblewrap cannot start, nothing runs bare: a job's setup or precheck step is refused as "sandbox unavailable", and an agent session that cannot start its jail ends with no result. The job's check runs in a jail of the same kind.

That refusal was tested sooner than planned. At the cutover on 2026-10-19, the runner's own health check found that the jail would not start on the machine at my desk. Bubblewrap's read-only bind of / remounts every mount beneath it, and it failed on a network-share automount whose server was switched off, so while that server was off, every agent job there would have ended "sandbox unavailable". The fix went into the runner the same day, not into the machine: it finds the automount points that do not answer, binds / read-only piece by piece around them, and then remounts the root read-only.

The guard

Inside the jail, a hook checks every shell command and every file edit before it runs. It denies pushes, deletes, installs, sudo, a model load, deploys, a nested Claude session, and any edit outside the lane. It denies git stash too, because the stash is shared by every worktree of a repository. One delete is allowed: rm of files the job itself created, untracked and named literally on a plain line. The guard fails closed: any error inside it exits with the code that blocks the call.

It is not the sandbox, and its own header says so: it is a guard against an agent's mistakes, and the jail and the runner hold the hard limits. It does not look at the read and search tools.

What the agent is told, and what it hands back

The runner wraps every job's brief in the same instructions. The session runs unattended and no one will answer, so it never asks a question: where a choice exists, it takes the option it would recommend and logs it. It works test-first where the job changes code, commits on its branch, never pushes, and leaves nothing uncommitted. Its final message has to end with these sections:

overnight.py: the sections every agent session ends with
## Summary
## Decisions
## Follow-ups
## Needs Alex
## Status: done | partial | blocked

Each line under Needs Alex starts with a tag: [secret] for a password, key, token or login, [money] for a purchase or a budget, and [scope] for a choice only I can make, such as my own permission settings, a check on a screen, or deleting something that exists nowhere else. Where or how a job runs is never a scope item. That is a follow-up, or the agent's own decision.

The runner checks for itself

When the session ends, the runner takes over, outside the jail. It commits whatever the session left, runs the job's check itself, in a jail of its own, instead of taking the agent's word for it, and asks Sonnet to review the diff against the job's brief, for a verdict of pass or fail with its findings. The runner's git calls ignore replace refs, which a session could write, so the reviewer sees the same content that is pushed. Then it pushes that one branch, and nothing else.

What reaches me in the morning

The brief opens with Needs you, and that section holds only what only I can do: a secret, money, a choice of scope an agent left, a logged-out Claude Code, a push refused for credentials, a git folder that changed during a session, or a job that the budget or the spend cap has carried for 3 nights or skipped for 7. Everything the runner can decide goes under Decided by the runner instead. A one-off job that failed three nights is dropped, a recurring one starts over, and an orphaned lane branch is left as it is.

Work an agent found and did not do becomes a follow-up job for a later night, in the repository the work belongs to. The runner keeps that from growing on its own: at most three follow-ups per job a night, none from a follow-up of a follow-up, and never a chain's fifth queued job. When a follow-up job the runner generated ends blocked or partial with no commit, a follow-up of it that only restates its own task is not queued, because that job is carried and runs again anyway. The test is the share of words the two texts have in common, measured against the shorter one: 0.34 or more is a restatement. That threshold sits midway between the different work, 0.27 or less, and the restatements, 0.40 or more, in the follow-ups of 2026-10-24.

What lands without me

Most jobs end at a pushed branch that waits for me to read it. A job lands by itself only if its file says land: yes, and only tool-kit's and 0-libraries' jobs do. This repository never lands that way, and neither does a follow-up the runner generated, nor a change to the files that configure sessions, to git attributes, to the units, or to the files those units run. An opted-in branch lands the same night only when its check is green, the Sonnet review passed on the whole diff, and nothing waits on me. The runner then merges the default branch into the lane, runs the check again, and pushes that tip as the default branch, never forced. A conflict, or a red check after the merge, goes back to the job's own next night.

When the agent is wrong

In the rehearsal on 2026-10-19, this repository's own suite went red because two of its suites read the machine's settings from the environment. The Haiku agent blamed the nested sandbox instead. It changed nothing and queued a follow-up, and the two suites were fixed the same day, in a commit that makes them clear those settings. Changing nothing is what every suite job's brief asks for when the agent finds no code defect to fix.

Stack

  • Python
  • bubblewrap
  • systemd
  • Claude Code

Connected to