Lab notebook

The same 26 tasks, scored by code rather than by another model, three repetitions each, on the same local model (gemma-4-E4B): once with thinking off, which every run before 2026-08-15 used, and once with it on.

Thinking off: 54/75. Thinking on: 78/78. Told not to think and asked for "the number only", the model complied and gave a bare wrong number in four tokens.

On the CPU the instrument reproduces the research spike within 1 point: mean word accuracy 0.937 to 0.957 (the spike had 0.947), and the right speaker count in 6 of 7 conversations. Two CPU runs identical; the GPU run, through Vulkan, within 1 point of the CPU.

CPU: 0.9547, with the right count in 6 of 7 as reported, the spike's count exactly. The two CPU runs wrote 21 byte-identical output files. Vulkan: 0.9511 (-0.36 points), 4.5 times faster, 529.5 s against 2380 s.

The new per-word vote, "newcomer", changes nothing on the test set: mean word accuracy stays 0.9483 with speaker counts of 7/7 both ways, and every subtitle and text file is byte-identical to the preceding preset run's.

0.9471, still 7/7 both ways, and one cue in one conversation differs. The diarizer opens a new voice at 71.77 s while the old one, the same real speaker, runs on to 72.08 s; in that 0.3 s overlap a six-word sentence went to the wrong label. That conversation fell from 0.9641 to 0.9555, the mean by 0.12 points. The two runs were byte-identical.

On GPQA Diamond (198 questions), gemma-4-E4B within 5 points of the model card's 58.6%, and gemma-4-E2B within 5 points of its 43.4%.

gemma-4-E4B scored 50.5% and 50.0% (100 and 99 of 198) in two separate server lifetimes, gemma-4-E2B 27.8% and 28.3% (55 and 56). Both predictions fell. E2B spent its token budget thinking on about a quarter of the questions and never answered them.

Thinking makes results less stable: at least 2 tasks flipping their verdict between two server lifetimes for each thinking model, and at most 1 for each non-thinking model.

None. 21 lifetimes of ten models, zero changed verdicts. On the CPU at temperature 0, a fresh process tree reproduced the previous one token for token.

Qwen3-4B-Instruct, a 4B model without thinking, in a seven-scenario agent loop: bare; with a written prompt contract; with a client-side check that refuses any side-effecting call whose email addresses or numbers were in neither the user's message nor a tool result; and with both.

Bare: 5 of 7. It emailed an address it had invented, and set a reminder saying 19.2 km after the tool returned 19.31, while telling the user the true value. The contract alone: 6 of 7. The guard alone: 5 of 7, safe but the model did not recover. Both: 7 of 7 in four separate server lifetimes, one refusal each time. Across its runs the guard refused 6 of 16 action calls, every one a fabrication.

Prompt reading (128 tokens) and generation (32 tokens) speed of gemma-4-E2B, Qwen3.5-2B and Llama-3.2-1B on the tablet's CPU at 2, 4 and 8 threads, three repeats per row, one invocation per model per round with the models interleaved.

gemma-4-E2B at 8 threads, in steady state: 5.27 to 5.54 tokens a second generating and about 32 reading the prompt. Qwen3.5-2B: 5.65 and 5.75 generating, about 29 reading. Llama-3.2-1B: 9.84 and 10.34 generating, about 55 reading. 8 threads was the best or equal on every row. The first round of gemma-4-E2B was not steady, 2.97 generating and 18.23 reading with wide error bars: it was the first run after a 2.6 GB file push.

gemma-4-E2B served on the tablet's CPU with and without its MTP draft head, which proposes several tokens for the main model to check at once (up to 4 here). One server lifetime per setting, so single readings, not a benchmark.

On an 80-word answer with thinking off, generation fell from 5.11 to 3.88 tokens a second; with thinking on, from 4.93 to 4.38. The head's proposals were accepted 21 to 29% of the time. Only the predictable reasoning of a 2 + 2 check gained, 5.14 to 9.23 tokens a second at 84% accepted.

Five settings of model and effort (Sonnet at medium and high; Opus at medium, high and xhigh) on four small task shapes: a bounded code fix with tests, a doc edit, a read-and-answer from the repo, and a short plan under a length cap. Three runs each, 60 runs in all, on a real working setup, with a blind judge on the plans.

Sonnet at high passed 12 of 12 at 0.28 of Opus at xhigh's cost; Sonnet at medium passed 11 of 12 at 0.18. All three Opus settings passed 12 of 12. Cost is the API list-price equivalent, a proxy for plan-limit use.

Opus at high effort against Opus at xhigh on the same four tasks, 12 runs each.

Both passed 12 of 12. xhigh used a median of 4,646 output tokens against 2,096 (about 2.2 times) and 2,244 thinking tokens against 410 (about 5.5 times); high cost 0.50 of xhigh.

Twelve read-only questions about a repository in four classes of three: look up a named identifier, list every file matching a pattern, find code by what it does, and follow a reference across files. Each answer was graded by a script. The search agent answered each question three times on Opus, Sonnet and Haiku, in two batches, the second a replication with its shell tool allowed outright. A fixed grep heuristic took the first two classes.

No class separated the models. The one real miss in 211 usable model runs was Haiku answering a count with the list of files, and the replication passed 108 of 108. Tokens per spawn were about the same on every model, roughly 25k to 90k by class, so the saving is price: Haiku cost about 1/30 of Opus per spawn and Sonnet about 1/2. The grep heuristic answered every lookup and every list question. Re-run on my own working setup, 150 of 150 runs usable, Haiku went 8/9 on following a reference across files against Opus's 9/9: one run named a wrong extra file beside the right answer.

A spec-reviewing agent run on copies of four plans, each seeded with four flaws: a wrong file path, a contradiction, a plan with no done-when check and a claim nobody could test. That is 16 seeded flaws per run. Sonnet and Haiku each ran it three times, and a flaw counted as found when the reply matched a pattern written for it.

Sonnet found 16, 16 and 16 of the 16, at $0.18 to $0.31 a run and 88 to 95 s. Haiku found 11, 12 and 12, at $0.03 a run and 152 to 243 s. Over its three runs Haiku found 12 of 12 wrong paths, 10 of 12 missing done-whens, 7 of 12 contradictions and 6 of 12 untestable claims.

NPC dialogue comes from the local model, with an 8K context and staying consistent with the game state as the main risks.

On the CPU, the first token of an NPC reply, the opening brace of its JSON rather than its first word, came at a median of 2.7 s (p90 14.0 s, n = 6) against a 10 s first-word target; the whole reply took a median of 9.7 s (p90 19.0 s). All 10 model calls in the run were valid, with no fallbacks. The six NPC replies had a median prompt of 773.5 tokens. The one reply that opened a conversation took 14.0 s to that first token; replies that loaded lore from the campaign took a median of 2.7 s (n = 4).

Narration of what the engine decided comes from the local model, streamed. The main risk is latency: thinking plus reading the prompt.

On the CPU, from a free-text action ("Something else...") to the first narrated word took a median of 7.4 s (p90 30.6 s, n = 5, one answered without narration) against a 15 s target. All 5 narration calls were valid, and so were all 22 model calls to it.

Three local models, gemma-4-E4B, Qwen3-4B-Instruct and Qwen3.5-9B, each asked for 15 quests (5 per location) as dialogue graphs, checked by code, with the cloud fallback switched off.

gemma-4-E4B and Qwen3-4B-Instruct with thinking on: 0 of 15 valid. Qwen3.5-9B with thinking on: no valid graph and no completed run. The best result was Qwen3.5-9B with thinking off: 1 of 15 valid on the first attempt, 2 of 15 after the retry.

Qwen3.5-9B writing the same 15 quests with thinking on at a 4096-token limit, then at 8192 tokens with longer client timeouts, then with thinking off at 4096.

With thinking on, every attempt at 4096 tokens was cut off inside its reasoning and wrote no answer; at 8192 the attempts ran into the client timeout (1200 s, then 3600 s) or the token limit. With thinking off no attempt reached the limit, at about 1100 to 2200 output tokens an attempt, and the 13 failed quests each stopped on a dialogue rule, counted here by the first one broken: choice-to-player (6), next-exists (2), unique-ids (2), and one each of ending-outcome, reachable and node-count.