How to read this page
Everything here I measured myself, on one 16 GB machine with an integrated GPU, except the tablet results, which come from an Android tablet. Each card is a shortened record from the repository. An expectation card shows what I wrote down before the run, what was measured, and what followed, and carries a stamp, WRONG or HELD. Where only the result was recorded, the card says what was tried and has no stamp, because I do not write an expectation after the fact. The lab notebook collects the cards from every project.
Scoring a model without a model
The judge instrument is 26 tasks whose answers a program can check, a runner for any OpenAI-compatible endpoint, and dated reports that are never overwritten. It is saturated: gpt-oss:20b and gemma-4-E4B with thinking both score 78/78, so it ranks the smaller models and nothing above them. Those 26 stay frozen as a regression gate. Ranking at or above the main model moves to a generated successor set of 268 items in seven families, run pairwise and on demand.
Published benchmarks, by my rules, choose candidates and nothing more. The model cards quote GPQA Diamond, 198 multiple-choice questions in physics, chemistry and biology, so I ran it on the machine, with the card's numbers written down first.
GPQA Diamond: how near the model card?
- Expected (written before the run)
On GPQA Diamond (198 questions), gemma-4-E4B within 5 points of the model card's 58.6%, and gemma-4-E2B within 5 points of its 43.4%.
- Measured
gemma-4-E4B scored 50.5% and 50.0% (100 and 99 of 198) in two separate server lifetimes, gemma-4-E2B 27.8% and 28.3% (55 and 56). Both predictions fell. E2B spent its token budget thinking on about a quarter of the questions and never answered them.
Verdict: WRONG- So
The comparison with the card is not like for like: the card's number is log-likelihood multiple choice on full-precision weights, while this was free generation with thinking on a 4-bit build. A log-likelihood run is still the only route to a number comparable with the card.
GPQA's authors ask that its questions are never posted. The repository therefore keeps scores, ids and settings, typed by hand, and never an item, and the full reports stay in a folder git ignores. In two separate server lifetimes gemma-4-E4B gave the same verdict on 191 of the 198 questions, so its own number is steady. The gap to the card is the gap between two different measurements, and the only route to a number comparable with the card is a log-likelihood run, which I have not done.
Small models as agents
The tool-calling lane asked one question: which model of ten billion parameters or fewer calls tools best for its size, on a CPU alone? It ran ten models in two lists, five that think and five that do not, against one fixed catalog of eight tools and 48 tasks in six categories. Every verdict is mechanical, and every failure carries a label: no call, wrong tool, wrong arguments, an extra call, an unparsed one. I committed the predictions before the first thinking model ran. One of them was that thinking makes a model less repeatable.
Does thinking make a model less repeatable?
- Expected (written before the run)
Thinking makes results less stable: at least 2 tasks flipping their verdict between two server lifetimes for each thinking model, and at most 1 for each non-thinking model.
- Measured
None. 21 lifetimes of ten models, zero changed verdicts. On the CPU at temperature 0, a fresh process tree reproduced the previous one token for token.
Verdict: WRONG- So
Falsified in the strongest way the instrument allows: there was no instability to compare. What thinking cost was time, 3 to 16 times the wall time per call against a non-thinking model of similar size.
So what thinking costs is time, not repeatability, and time decides which model can run an agent on a CPU. On 2026-09-23 I decided the lane's answer. The primary agent model is gemma-4-E4B with its MTP draft head, a second small model that proposes several tokens for the main one to check at once. The secondary, for large or unfamiliar tool catalogs, is Qwen3.5-9B. Qwen3-4B-Instruct takes high-volume, low-stakes steps, but only behind a written contract and a guard in the client. Agent steps run at temperature 0.
The fast model, Qwen3-4B-Instruct, is also the one that makes things up. Left bare in an agent loop it invents values. The contract and the guard were my two candidate fixes, and I measured each alone and both together.
Guarding a small model's tool calls
- Tried
Qwen3-4B-Instruct, a 4B model without thinking, in a seven-scenario agent loop: bare; with a written prompt contract; with a client-side check that refuses any side-effecting call whose email addresses or numbers were in neither the user's message nor a tool result; and with both.
- Measured
Bare: 5 of 7. It emailed an address it had invented, and set a reminder saying 19.2 km after the tool returned 19.31, while telling the user the true value. The contract alone: 6 of 7. The guard alone: 5 of 7, safe but the model did not recover. Both: 7 of 7 in four separate server lifetimes, one refusal each time. Across its runs the guard refused 6 of 16 action calls, every one a fabrication.
- So
Neither fix works alone, so they ship together. The confirmation a user reads is rendered from the call that was executed, never from the model's summary: its summary was right both times its action was wrong.
The guard lives in the client, not the model. Before any call with a side effect runs, each email address and each number in its arguments must appear in the user's message or in an earlier successful tool result. A call that fails is not executed, and the model is told to use the tool's value, look it up or ask. The two thinking models never tripped it. The confirmation a user reads is drawn by the client from the call that actually ran, because the small model's own summary was right both times its action was wrong.
Who said what, in Romanian
I wanted speaker labels on whisper transcripts of Romanian recordings, P1: and P2: in front of each line, running on the CPU. Whisper cannot do it alone. Its own option compares left and right channel energy, and a recording from one microphone has identical channels. Labels need a second model, a diarizer, run on the same audio and merged with whisper's words by time.
There was no Romanian test set with known speakers, so I built one from VoxPopuli's Romanian test split: real speakers from the European Parliament, under a CC0 licence, cut into conversations of two to four speakers whose turns are known by construction. On it the diarizer I chose gave 94.7% of words the right speaker on the single-session set and 93.5% on the harder cross-session one, with the same speaker counts and the same mean accuracy on the CPU and the GPU. A fully labelled transcript runs at about 1.2 times real time on the CPU. The test set and its scorer then became an instrument, and the first thing it had to show was that it agreed with the research it came from.
Speaker labels: does the new instrument match the spike?
- Expected (written before the run)
On the CPU the instrument reproduces the research spike within 1 point: mean word accuracy 0.937 to 0.957 (the spike had 0.947), and the right speaker count in 6 of 7 conversations. Two CPU runs identical; the GPU run, through Vulkan, within 1 point of the CPU.
- Measured
CPU: 0.9547, with the right count in 6 of 7 as reported, the spike's count exactly. The two CPU runs wrote 21 byte-identical output files. Vulkan: 0.9511 (-0.36 points), 4.5 times faster, 529.5 s against 2380 s.
Verdict: HELD- So
All four predictions held. This run became the baseline: later changes to the labelling were each predicted relative to it, before they ran.
That run became the baseline, and each later change to the labelling was predicted against it before it ran. One of those predictions failed in a small way.
Speaker labels: a one-cue miss
- Expected (written before the run)
The new per-word vote, "newcomer", changes nothing on the test set: mean word accuracy stays 0.9483 with speaker counts of 7/7 both ways, and every subtitle and text file is byte-identical to the preceding preset run's.
- Measured
0.9471, still 7/7 both ways, and one cue in one conversation differs. The diarizer opens a new voice at 71.77 s while the old one, the same real speaker, runs on to 72.08 s; in that 0.3 s overlap a six-word sentence went to the wrong label. That conversation fell from 0.9641 to 0.9555, the mean by 0.12 points. The two runs were byte-identical.
Verdict: WRONG- So
Shipped anyway: +3.9 points of units on the overlaps of a real two-person recording against -0.12 points here. The earlier "changes nothing" had been measured on the default settings, whose turns barely overlap; the offline preset overlaps more.
I shipped the change. The rule for a word that two turns share, which goes to the newer voice, was chosen on a recording I labelled by hand: of the eight sentences the two candidate rules split, the newer-voice rule got seven right and longest overlap got one. The instrument keeps its own gate: its standard run exits with an error below 0.94 word accuracy, so a regression cannot pass quietly.
Small models on an Android tablet
llmctl also runs on Android, under Termux, on llama.cpp's CPU build. That build has no router, so the llama.cpp loader has a single mode there: one server for one model, and a second model is a restart. I measured what small models do on the tablet's CPU, with the models interleaved so that every round tests all of them.
Small models on an Android tablet
- Tried
Prompt reading (128 tokens) and generation (32 tokens) speed of gemma-4-E2B, Qwen3.5-2B and Llama-3.2-1B on the tablet's CPU at 2, 4 and 8 threads, three repeats per row, one invocation per model per round with the models interleaved.
- Measured
gemma-4-E2B at 8 threads, in steady state: 5.27 to 5.54 tokens a second generating and about 32 reading the prompt. Qwen3.5-2B: 5.65 and 5.75 generating, about 29 reading. Llama-3.2-1B: 9.84 and 10.34 generating, about 55 reading. 8 threads was the best or equal on every row. The first round of gemma-4-E2B was not steady, 2.97 generating and 18.23 reading with wide error bars: it was the first run after a 2.6 GB file push.
- So
8 threads is the setting. gemma-4-E2B generates as fast as Qwen3.5-2B from a file twice the size, because not all of its per-layer embeddings are read for each token. The cold round stays on the record, and the steady rounds set the numbers.
The cold first round stays on the record with its wide error bars. A suspiciously wide bar on one row is how I know the machine was not in steady state: it is a signal, not noise to average away.
On my 16 GB machine, a draft head like the agent model's is the largest software speed-up I have measured. Does it help on the tablet?
A speed-up that did not travel
- Tried
gemma-4-E2B served on the tablet's CPU with and without its MTP draft head, which proposes several tokens for the main model to check at once (up to 4 here). One server lifetime per setting, so single readings, not a benchmark.
- Measured
On an 80-word answer with thinking off, generation fell from 5.11 to 3.88 tokens a second; with thinking on, from 4.93 to 4.38. The head's proposals were accepted 21 to 29% of the time. Only the predictable reasoning of a 2 + 2 check gained, 5.14 to 9.23 tokens a second at 84% accepted.
- So
On a 16 GB machine with an integrated GPU the same head is the largest software win, about 1.55 times. It earns its place there and not on the tablet's CPU. Draft lengths below 4 were not tried.
These are single readings, one server lifetime for each setting, and I have not tried draft lengths below four. So I do not claim the head is useless on the tablet, only that this setting does not help there.
What I have not measured
The marks make this list easy to write. Runs of hours still go without the draft head on the machine, because whether a newer build lets them use it again is not measured. A GPQA number comparable with the model card needs a log-likelihood run. Draft lengths below four on the tablet are untried. Until I run them, each of these stays marked as read, not measured.