What I measured before I trusted a model

Brains, my agent memory system, pulls tickers and entities out of every page it ingests. The local model it used, Qwen 3 30B, mostly did the job, but its output sometimes failed to parse as JSON, and a regex fallback was catching the misses. Every fallback is a silent guess.

So I ran both candidates on twelve real pages from the feed, with the task, prompt and settings the system actually uses: JSON mode, temperature zero, each model warmed once before timing. Gemma 4 26B returned valid JSON on 100% of pages against 83%, and found 102 tickers against 79, at nearly the same speed: 70.7 against 72.4 tokens a second. The switch was one configuration line, and reversible.

Two months later I asked the next question: does a bigger build of the same model help? Thirty pages, with the 8-bit build as the reference. The 4-bit build agreed with it at F1 0.997, missing one ticker in thirty pages and adding none, and ran at 69.7 tokens a second against 51.8. It stayed.

The write-up says plainly what that second test cannot show. It scores agreement between two builds, not correctness: two builds wrong in the same way would look identical. It answers whether more precision changes the extraction, not whether the extraction is right.

Brains used to write a daily digest back into its own memory. A digest is a dense summary of exactly the topics the next digest will ask about, so the digests started outranking the pages they summarised. The agents were reading their own summaries as evidence.

Before changing anything I wrote fifteen golden queries: the queries the agents really issue, each with the pages it should find. A script scored hit@1, 5 and 10, mean reciprocal rank, poison (synthesised pages in the top ten) and freshness (the median age of the top five, for time-sensitive queries).

The fix kept synthesised pages out of retrieval by default and gave queries a recency stage. Poison hits went from 25 to 0, and the twelve affected queries to none. The median age of time-sensitive results went from 15 days to 10.5.

Ranking held: MRR 0.844 to 0.843, hit rates unchanged. That is the number I watched hardest. A fix that removes the poison by breaking the ranking is not a fix.

Brains also builds a knowledge graph. An early pass let a local model propose relations between entities in any terms it liked. It wrote 1,063 edges across 350 different predicate types. Reading them, I found a chip maker recorded as the chief executive of a software company.

I hand-checked a sample of 25. Under a gate that demanded a verbatim quote, 28% held up. A local judge model scored the same sample at 45%, and the best judge, Claude, at about 56%. None came near the 90% I want before a graph is trusted, so I deleted all 1,063, after writing each one to a log so it could be restored.

The replacement is stricter by design: 14 defined relations, each with a direction, instead of 350 open ones. The model must return the sentence that states the relation, and code checks that the quote is verbatim on the page, that both entities are named in it, and that the verb fits. Precision over recall: a doubtful edge is dropped.

And it writes nothing yet. The pass runs in shadow mode every night, and goes live only when 50 human labels show at least 90% precision, reported with a Wilson 95% interval so a small sample cannot pass for a result. As of November 2026 it has no labels, so the gate is closed. I would rather show a closed gate than an open one I cannot defend.

Breakwise, the market-intelligence desk I am building, reads company filings overnight and stores a fact only with the sentence it came from. A model reviews those facts before anything reaches a thesis. The reviewer is itself a model, so it gets the same treatment.

Each reviewer has a golden set built from the desk's own past mistakes: a figure filed under the wrong company, a range stored as a single point, a debt raise stored as an order. Clean controls sit beside them, which the reviewer must leave alone. The bar is recall of at least 0.8 on the mistakes and a false-flag rate of at most 0.2 on the controls, and only a recorded, passing run switches a reviewer on.

The first reviewer through ran as a chain: a local model reads every item, and the frontier model reads only what the local model flags. On the golden set it caught 10 of 10 mistakes and flagged none of the 9 controls, and the frontier model read 11 of the 19 items.

That repository is private, so this case you have to take from the description. The method is the same as the three public ones above, and the Breakwise case study has the rest.

None of this is exotic. A golden set is a list of real cases with the right answers. An A/B is two runs on the same inputs. Shadow mode is a pass that writes to a log instead of the system. What they share is an order of operations: the measurement exists before the change, the result is committed next to the code, and the gate is set before the run, not after it.

It is the discipline I learned on physical systems. At NTU, any past result of the haze detector I built could be regenerated by replay, and the February 2026 replay is the number I stand on: ten alerts where the old version issued none, the first nearly six hours before the haze reached breathing height. A model that reads filings deserves no less than one that reads a LiDAR.