quest-builder

The problem

I built quest-builder to write the quests for hand-written text worlds, one world per project. A world is a set of files: its locations, the characters and items in each, and which places connect. What it lacks is the quests, each with an offer dialogue and a completion dialogue written as a branching graph that a program can load, not just prose a person can read.

The World screen of the quest-builder window, showing a hand-written coast world: a table of its locations with their connections, characters and items, and a per-location setting for quests and for dialogue branches. — A world as the tool reads it: each location with its connections, characters and items, and its own settings for how many quests to write and whether the dialogue branches.

Programs and people both read the quests, so the tool writes one project-neutral JSON format and leaves each game's own format to a small exporter. The local models are served by 1-llm; the fallback is Ollama Cloud. quest-builder does not start or swap models yet: when an endpoint serves a different model from the one a step wants, it prints the llmctl command that would serve the right one, and stops.

The pipeline

The pipeline has five steps. The run command covers the first four for each location of a world and writes the results out; the blind comparison is a command of its own, compare.

Generate. The location's data fills a prompt template, and a model writes a quest in the neutral format.
Code-check. Code does every check that code can: names, fields, lengths and the structure of the dialogue graph.
Critique. A model reviews the quest against a checklist, given the location and the code's findings, and answers ok or revise with its issues. The model judges only what code cannot.
Revise. A new version fixes only the issues the critique raised, and the code check runs on it. A revision that fails the check, when the version before it passed, is dropped, and the earlier version is the one exported.
Blind A/B. Each quest's first draft and its final version are shown as A and B in a random order. I answer a, b, tie or skip, and only after the verdicts are recorded does the tool reveal which was which, with the tally. That is how I find out whether revising helps, instead of assuming it does.

The pipeline from the command line

# load and check the world; no model
questctl/questctl.sh --project _trial world

# generate, check, critique, revise, export
questctl/questctl.sh --project _trial run

# the latest run, per quest
questctl/questctl.sh --project _trial status

# blind A/B: first draft against final
questctl/questctl.sh --project _trial compare latest

Every run keeps each attempt's prompt, raw answer and check findings on disk, so a failure can be read afterwards instead of guessed at.

The Results screen of the quest-builder window: a table of the quests in one run, and below it one quest's first draft and its revised final version side by side, each with its check result, its critique and its two dialogue stages drawn as graphs, a dashed arrow marking a loop. — Results: a quest's first draft beside its revised final version, with the check findings, the critique and each dialogue stage drawn as a graph. A dashed arrow marks a loop.

The Compare screen of the quest-builder window: two quest texts side by side, labelled A and B, with buttons to say A is better, B is better, a tie or skip. — The blind comparison. A and B are a first draft and its final version in random order, and the screen does not say which is which until the verdicts are recorded.

Past those steps, a run's final versions can be exported into a game's own format, and only those, never a newer unchecked version. The one target so far is LLM Dungeon, which imports a quest pack of each quest's title, hook, goal and the item to bring out. A quest that would break the game's limits, or that names no item, is skipped, with the reason given.

What the local models could do

The design said to test the local models first and make the one that writes valid graphs the default, and to send the quests that still fail to a cloud model. So before building on any model, I ran one fixed trial world through each candidate. Every run used the same world, five quests per location and the cloud fallback switched off, so a failure stays a failure instead of being rescued.

Can a local model write valid quest dialogue graphs?

2026-10-26

Tried: Three local models, gemma-4-E4B, Qwen3-4B-Instruct and Qwen3.5-9B, each asked for 15 quests (5 per location) as dialogue graphs, checked by code, with the cloud fallback switched off.
Measured: gemma-4-E4B and Qwen3-4B-Instruct with thinking on: 0 of 15 valid. Qwen3.5-9B with thinking on: no valid graph and no completed run. The best result was Qwen3.5-9B with thinking off: 1 of 15 valid on the first attempt, 2 of 15 after the retry.
So: No local model writes dialogue graphs reliably under these settings. Quests that fail go to the cloud fallback, and the local default is Qwen3.5-9B with thinking off.

The best candidate is the default now, and it still writes a valid graph on the first try for 1 quest in 15. No local model writes these graphs reliably under these settings, so the quests that fail go to the cloud fallback the design had planned. A quest whose graph the local model breaks again after the retry is written again in the cloud. Only a broken dialogue rule sends work there: a timeout or a server error never sends work, and cost, to the cloud.

The early zeros were not all the same failure. The first two candidates wrote graphs that broke rules. Qwen3.5-9B, with thinking on, never got as far as an answer.

Thinking on or off when writing a graph

2026-10-26

Tried: Qwen3.5-9B writing the same 15 quests with thinking on at a 4096-token limit, then at 8192 tokens with longer client timeouts, then with thinking off at 4096.
Measured: With thinking on, every attempt at 4096 tokens was cut off inside its reasoning and wrote no answer; at 8192 the attempts ran into the client timeout (1200 s, then 3600 s) or the token limit. With thinking off no attempt reached the limit, at about 1100 to 2200 output tokens an attempt, and the 13 failed quests each stopped on a dialogue rule, counted here by the first one broken: choice-to-player (6), next-exists (2), unique-ids (2), and one each of ending-outcome, reachable and node-count.
So: Thinking spends the budget before any answer, so it is off for graph writing. What fails now is a named rule break rather than an empty answer.

With thinking off, the failures are named rules, so they can be counted and grouped. The next question was whether deterministic repairs in plain code could fix them before any model is involved. I replayed such repairs offline over the graphs that failed. Renaming a repeated id, pointing a dangling next at the closest existing id and deleting nodes that nothing reaches each cleared their rule in every quest where it occurred. The closest id is chosen by spelling, so it can be the wrong node, and nothing in the rules checks meaning. No quest passed afterwards. choice-to-player, which none of the repairs touches, still broke in 11 of the 13 failed graphs, and it was the only rule left in 4 of them. Nine kept another broken rule, six of them an ending-outcome or offer-accept rule. A repair that fills in a missing ending outcome from keywords mostly left its rule standing. I did not extend its keyword list after seeing the failures, since that would tune it to the same quests it was scored on. Nothing from the replay is wired into generation.

Three front ends over one core

The core is questctl, a Python package and command line that uses only the standard library. The other two front ends hold no pipeline logic of their own.

The command line, questctl, does the work.
A terminal menu, quest.py, in plain numbered menus. It reads through the core's library and acts only by running questctl commands, showing each command before it runs and its exit code after.
A desktop window, over the same read layer and the same commands: projects, the world, a run, the results, the blind comparison, endpoints and settings. One job runs at a time, its output streams to a log pane, and every change shows up there as a command. It is pywebview on Qt WebEngine, the only third-party code in the repository, pinned with a hash for every wheel. Its page is plain files loaded from disk, talking to Python through pywebview's bridge: no HTTP server and no port.

Because the menu and the window only ever run questctl commands, behaviour lives in one place and the three cannot drift apart. The core's tests run against a scripted fake endpoint, so they need no model, and the window's page is tested in headless Chromium through Playwright, over its real bridge.

How I built it

I built quest-builder as AI-orchestrated engineering, measured and reviewed. It started as a written design, with each question I had answered recorded in it, and then a plan for each part. Claude Code co-authored most of the commits against those plans. The default model for writing a quest, and thinking off, come from the trial above. The models for critique and revise, and the cloud fallback model, are still starting values that I have not measured.