Downloads · 30 days
70
6% of all-time downloads
pngwn/system-one-qwen3.5-4b-scorer
system-one-qwen3.5-4b-scorer is a machine learning model from pngwn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as cc-by-nc-4.0.
A single-pass "System One" decision model in the shape of TypeSafe's Jev: it takes unstructured state plus a set of typed questions (yes/no, Choice, numeric Score) and returns a probability distribution over exactly t…
Downloads · 30 days
70
6% of all-time downloads
All-time downloads
1.3K
Public
Repo size
142 MB
Likes
21
Public
Click a slice to open those files.
.safetensors122 MB · 86%
From the Hugging Face model README
A single-pass "System One" decision model in the shape of TypeSafe's Jev: it takes unstructured state plus a set of typed questions (yes/no, Choice, numeric Score) and returns a probability distribution over exactly the options the caller supplied — in one forward pass, with no autoregressive generation.
Each (state, question, option) triple is scored by a sequence-classification head and the per-question logits are softmaxed. Because the output space is the option set, the output is type-safe by construction rather than by post-hoc parsing: there is no token stream that can drift outside the schema.
Training code: system_one.py in this repository.
Route, classify, prioritise or score a decision whose options you already know. This is not a chat model and not a generator — it cannot produce free text.
Held-out test split, option sets uncapped at evaluation, temperature 1.75 fitted on val.
| task | n | accuracy | ECE | Brier |
|---|---|---|---|---|
| ag_news | 64 | 0.922 | 0.021 | 0.131 |
| banking77 | 64 | 0.891 | 0.049 | 0.176 |
| go_emotions | 64 | 0.859 | 0.062 | 0.202 |
| tickets_language | 64 | 0.891 | 0.069 | 0.189 |
| tickets_type | 64 | 0.750 | 0.052 | 0.329 |
| mmlu | 64 | 0.703 | 0.188 | 0.407 |
| yelp_score | 64 | 0.641 | 0.117 | 0.460 |
| tickets_priority | 64 | 0.469 | 0.058 | 0.612 |
| tickets_queue | 64 | 0.234 | 0.244 | 0.848 |
| ALL | 576 | 0.707 | 0.044 | 0.373 |
Validation split, all 9 families, per-task cap 64 (n=537): accuracy 0.752, ECE 0.032.
| model | accuracy | ECE | Brier |
|---|---|---|---|
| this model, raw head (test, n=576) | 0.707 | 0.135 | 0.415 |
| this model, T=1.75 fitted on val (test, n=576) | 0.707 | 0.044 | 0.373 |
| prompted Qwen3.5-4B-Base, 26-letter answer (val, n=112) | 0.679 | 0.093 | 0.439 |
Temperature scaling cuts ECE from 0.135 to 0.044 — about 2× better calibrated than the prompted baseline, at identical accuracy. The raw head is overconfident; that gap is the whole reason a calibration stage exists.
The baseline row is not measured on the same rows: it covers 7 of 9 families (112 questions, 16 per family), because banking77 (77 options) and ticket routing (52) do not fit its 26-letter answer alphabet. On those 7 families this scorer averages 0.748 test accuracy against the baseline's 0.679.
112.3 ms per question at 4 options — one forward pass, option batch scored together. A smoke run measured 559.7 ms at 77 options. Latency scales with option count and sequence length, not with output length, because nothing is generated.
pngwn/system-one-decisions — 12,913 train questions across 9 task familiesscore head
(30.5M trainable of 4.24B)tickets_queue is the visible casualty at 0.234
accuracy, and it is also the worst-calibrated task (ECE 0.244).yelp_score are the weakest non-routing tasks.