Downloads · 30 days
0
pngwn/nanodiff-350m-typed-decisions-lam1
nanodiff-350m-typed-decisions-lam1 is a machine learning model from pngwn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as mit.
Proper-scoring arm (λ = 1.0) of the typed-decision calibration study: identical to the control except the decision-slot CE adds to the loss (a log-score — proper scoring rule — on the answer token). See the full report.
Downloads · 30 days
0
Access
Public
Updated Sep 16, 2026
Repo size
9.2 GB
Likes
0
Public
Click a slice to open those files.
.pt1.5 GB · 100%
From the Hugging Face model README
Proper-scoring arm (λ = 1.0) of the typed-decision calibration study: identical to the control except the decision-slot CE adds to the loss (a log-score — proper scoring rule — on the answer token). See the full report.
eval_calibration_step2000.json here) — the arm overtrains past ~2k stepsnanodiff-350m-typed-decisions-lam1.pt (step 3000) + the step-2000 revision (commit abf2adb715; torch.load → state dict; load exactly as in code/eval_calibration.py)eval_calibration.json (+ logits), eval_calibration_step2000.json (+ logits)| group | n | acc | ece_hard | ece_soft | ece_soft|temp | AUROC | [email protected] |
|---|---|---|---|---|---|---|---|
| ALL | 5214 | 0.676 | 0.324 | 0.058 | 0.048 | 0.915 | 0.957 |
| choice (1-of-10) | 1195 | 0.122 | 0.878 | 0.195 | 0.084 | 0.518 | 0.127 |
| noul | 231 | 0.662 | 0.338 | 0.188 | 0.088 | 0.662 | 0.793 |
| severity (Score) | 338 | 0.713 | 0.287 | 0.048 | 0.083 | 0.800 | 0.929 |
| workflow4 (k=4) | 2436 | 0.858 | 0.142 | 0.020 | 0.073 | 0.875 | 0.998 |
| group | n | acc | ece_hard | ece_soft | ece_soft|temp | AUROC | [email protected] |
|---|---|---|---|---|---|---|---|
| ALL | 5214 | 0.669 | 0.331 | 0.036 | 0.027 | 0.919 | 0.953 |
Headline: the λ term improves soft calibration from the control's 0.065 to 0.036 at its best
step (9× better than the hard-confidence reading 0.331, at equal accuracy), and k=4 multi-slot
scoring stays near-perfectly trustworthy (ece_soft 0.020, [email protected] 0.998). On severity the posterior
is within L1 0.072 of the closed-form Bayes-optimal posterior (model acc 0.713 vs oracle 0.716).
Caveats: choice is at chance (AUROC ≈ 0.52) — a knowledge limitation of the 350M base;
escalate/review gold labels disagree with their own Bayes oracle; see the report.
# Only the answer positions are [MASK]ed. See code/eval_calibration.py (pinned at 65691f24)
# for the exact released path (loading, option-token map, T scaling).
logits = model(x_masked, t=...) # (B, 512, 50304)
probs = logits[:, slot_idx].softmax(-1) # the decision distribution
Control arm (λ=0): nanodiff-350m-typed-decisions-lam0.