typical-large-preview

typical-large-preview

A 14B research checkpoint that did not clear its own release bar. Published anyway, with the numbers that failed it.

This is the strongest model the Typical project has produced on most measures, and it is labelled a preview because it misses one term of a pass rule we wrote down before training it. The full reasoning is below. If you want a model to deploy, use typical-small or typical-medium. If you want to see how far this architecture gets at 14B, this is it.

pip install typical-ai

from typical_ai import Typical

m = Typical.from_pretrained("OzLabs/typical-large-preview")

m.choice(state, "What does the customer want?", ["refund", "replacement", "repair"])
m.noul(state,   "Is the order still under warranty?")
m.score(state,  "How urgent is this ticket?", ["0", "1", "2", "3"])

The import is typical_ai, not typical (that name belongs to an unrelated PyPI package). This checkpoint uses the structured "semif" render, which the package reads from the checkpoint automatically — you do not configure it.

⚠️ Non-ASCII input: upgrade to typical-ai 0.1.1

Versions of typical-ai up to and including 0.1.0 ASCII-escaped the criterion and option descriptions before the model saw them, so any non-Latin text — Hebrew, Arabic, Russian, Greek, Chinese, or an English rubric quoting a name in another script — arrived as \uXXXX, six ASCII characters per letter. Predictions on such rubrics were made on unreadable input, and the prompt grew about 9x, which could raise AssertionError: suffix > max_suffix=1024.

pip install -U "typical-ai>=0.1.1"

The weights are unchanged — re-download nothing. This was a bug in the renderer, not the model. ASCII prompts are byte-identical across the two versions, so every English result, and everything this checkpoint was trained on, is unaffected. This model uses the semif render and was affected; typical-small-preview and typical-medium render letters_nonull and were not. Details: CHANGELOG.

Found by running the published checkpoints against JevBench-HE, a Hebrew decision benchmark. After the fix, Hebrew's real rendering overhead is 1.1–1.3x English, not 9x.

The release bar, and how this model did against it

Written down before training, in PLAN7.md:

(JevBench hard ≥ .559 OR hard Brier ≤ .65) AND long-document policy accuracy ≥ .35

So it fails. We are publishing it as a preview rather than quietly dropping it, for a reason that is itself the most interesting result we have.

That .053 comes from a 19-item benchmark family we no longer trust. Across two seeds of an otherwise identical matched comparison, that same family gave 6/19 against 2/19, and then 5/19 against 5/19 — a four-item gap, then exactly zero. It cannot distinguish a real effect from nothing at that sample size.

So we built a bigger test: 605 held-out long-document policy states, leak-checked against the training corpora, scored with no truncation. This model gets .974 on it. That is the capability the failing term was trying to measure, and it passes overwhelmingly.

We are not going to quietly rewrite a pass rule after seeing the result. The rule as written is failed, so this ships as a preview. The rule as intended is passed. Both statements are true and you should have both.

Results

All JevBench numbers are a public-subset run (72 standard / 48 easy / 111 hard public ids), not a ranked leaderboard entry. Intervals matter more than point estimates here: hard is ±9 points at n = 111 and standard is ±6–13 at n_eff = 36 (72 items, but only 36 independent states, each appearing as two paraphrases).

It is the best 14B we trained on 10 of 16 measured axes, including every knowledge and NLI number and the best hard-tier calibration in the project. The 320-way choice result is the one we find most striking: .858 against .529 for the prior recipe, on candidate sets far larger than anything in training.

What is wrong with it

A frozen Qwen3-14B with three examples in its prompt beats every trained 14B we have on the hard tier, .559 against .468 here. On a paired per-item test against our tl1b run that gap is +.108 [+.027, +.189], p = .015, so it is not a sampling artefact. Teaching a model to emit direct decisions improves the target distribution and costs capability outside it. We think this says the training recipe is the bottleneck rather than the architecture, but we have not proven that, and it is the clearest open problem in the project.

The hard tier is unsolved at every size. Long-document, multi-step, temporal, unit and trade-off decisions sit far below the standard tier for every model we have trained, and scaling the recipe has not fixed it.

Standard-tier differences among our 14B runs are not resolvable. .903 here against .931 for tl1b is well inside the interval. Do not read the ordering as meaningful.

Rendering is a confound in every cross-model comparison, including ours. The same frozen checkpoint moves 12.5 points on JevBench standard from the render alone, with no training. Benchmark score is capability plus interface compatibility, and the field's standard-tier numbers should be read with that in mind — ours included.

Details

Qwen3-14B-Base truncated at tap 28 of 40, LoRA r16 on the top 8 kept layers, structured semif render, factored abstention gate, ordinal-smoothed Score targets, per-row Bernoulli Noul head, 3,072-token decision states with --drop_truncated, 8,000 steps, checkpoint selected on uncertainty + curriculum validation NLL rather than accuracy. Single decision at K=2 / 32 / 256: 60 / 70 / 281 ms on one H100 (the semif render's per-option JSON is verbose, which is what costs the high-K figure).

Weights are a LoRA adapter plus small Choice/Score/Noul heads over the Apache-2.0 backbone. Training data licenses and the full experimental record, including everything that did not work, are in the public repo: https://github.com/GuyNachshon/typical.

Lineage

typical-small-preview → typical-small → typical-medium → typical-large-preview (this model, withheld from full release). Sibling 14B runs tl1b, tl1b_nokd and ladder_14b are described in the repo's REPORT.md §3ah–§3ak.