Downloads · 30 days
0
pngwn/nanodiff-350m-typed-decisions-lam0
nanodiff-350m-typed-decisions-lam0 is a machine learning model from pngwn. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as mit.
Control arm (λ = 0.0) of the typed-decision calibration study: LLaDA-style SFT fine-tune of Sebasdi/nanodiff-350m-base on pngwn/typed-decisions with the decision-slot CE computed and logged but not added to the loss (…
Downloads · 30 days
0
Access
Public
Updated Sep 16, 2026
Repo size
9.2 GB
Likes
0
Public
Click a slice to open those files.
.pt1.5 GB · 100%
From the Hugging Face model README
Control arm (λ = 0.0) of the typed-decision calibration study: LLaDA-style SFT fine-tune of
Sebasdi/nanodiff-350m-base on pngwn/typed-decisions with the decision-slot CE computed and
logged but not added to the loss (correctness-only training). See the full report.
nanodiff-350m-typed-decisions-lam0.pt (1.5 GB, torch.load → state dict; load exactly as in code/eval_calibration.py)eval_calibration.json, eval_calibration_logits.npz| group | n | acc | ece_hard | ece_soft | ece_soft|temp | AUROC | [email protected] |
|---|---|---|---|---|---|---|---|
| ALL | 5214 | 0.667 | 0.333 | 0.065 | 0.041 | 0.922 | 0.957 |
| choice (1-of-10) | 1195 | 0.110 | 0.890 | 0.176 | 0.088 | 0.526 | 0.119 |
| noul | 231 | 0.606 | 0.394 | 0.208 | 0.102 | 0.681 | 0.716 |
| severity (Score) | 338 | 0.704 | 0.296 | 0.053 | 0.081 | 0.838 | 0.947 |
| workflow4 (k=4) | 2436 | 0.852 | 0.148 | 0.023 | 0.060 | 0.880 | 0.998 |
Headline: the masked-slot softmax (ece_soft 0.065) is 5× better calibrated than the
argmax-with-confidence-1.0 reading of the same predictions (ece_hard 0.333), at identical accuracy.
On synthetic severity the posterior is within L1 0.077 of the closed-form Bayes-optimal
posterior (model acc 0.704 vs oracle 0.716; bayes_ece 0.030).
Caveats: the choice domain is at chance (AUROC 0.526) — a knowledge limitation of the 350M
base, not a calibration artifact. escalate/review gold labels disagree with their own Bayes
oracle; see the report.
# Only the answer positions are [MASK]ed. See code/eval_calibration.py (pinned at 65691f24)
# for the exact released path (loading, option-token map, T scaling).
logits = model(x_masked, t=...) # (B, 512, 50304)
probs = logits[:, slot_idx].softmax(-1) # the decision distribution
Sibling arm (λ=1.0): nanodiff-350m-typed-decisions-lam1.