Downloads · 30 days
26
9% of all-time downloads
25b3nk/smollm2-360m-boolq-calibration-grpo2-v1
smollm2-360m-boolq-calibration-grpo2-v1 is a text generation model from 25b3nk. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
GRPO-trained checkpoint from slm-calibration-rlcd — a project training small LMs to make calibrated yes/no decisions, inspired by TypeSafe AI's "Jev" model / "RLCD" (Reinforcement Learning for Calibrated Decisions). F…
Downloads · 30 days
26
9% of all-time downloads
All-time downloads
287
Public
Parameters
362M
1.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.4 GB · 100%
From the Hugging Face model README
GRPO-trained checkpoint from slm-calibration-rlcd — a project training small LMs to make calibrated yes/no decisions, inspired by TypeSafe AI's "Jev" model / "RLCD" (Reinforcement Learning for Calibrated Decisions). Full project: 25b3nk/slm-calibration-rlcd.
This is the full GRPO-trained model, built on top of sft-warmup-v2. It is a real but modest calibration improvement, not a finished or perfectly calibrated model — see the eval table and discussion below before over-reading the result.
Three earlier GRPO runs on top of the original SFT checkpoint (sft-warmup-v1) all collapsed: the model's confidence output converged to just 1-2 of the two fixed confidence-target constants used in v1's SFT data (0.92 and/or 0.65), regardless of training steps or learning rate. That's not calibration — it's the model parroting the SFT heuristic. Root cause: v1's SFT prior had near-zero output entropy on the confidence dimension, so GRPO had nothing to diversify from.
The fix was not a GRPO hyperparameter change — it was upstream, in SFT: v2 (see sft-warmup-v2) samples confidence targets from ranges instead of two fixed constants, and its prompt explicitly asks for a confidence value. This checkpoint (grpo2-v1) reuses the exact same GRPO hyperparameters as the failed v1 attempts (lr=1e-6, confidence-alignment weight λ=0.5, KL penalty β=0.04, num_generations=6, 1 epoch, 56 steps) on top of sft-warmup-v2, specifically to isolate the SFT/prompt change as the cause of the fix.
It worked: this checkpoint produced 11 distinct confidence values across the 500-example eval set (spanning roughly 0.61-0.89), versus 1-2 distinct values for every prior GRPO attempt. That's the key qualitative result — genuinely non-degenerate, varied confidence output for the first time in this project.
A correctness-only control run (grpo2-control-v1: same pipeline and hyperparameters, but the confidence-alignment reward term zeroed) was also trained for comparison. The treatment (this checkpoint) beats the control on both calibration metrics at equal accuracy, which validates that the calibration-specific reward terms are doing real, measurable work rather than just adding noise.
Given a BoolQ-style passage + yes/no question, outputs:
<answer>yes|no</answer><confidence>0.XX</confidence>
1[answer==label] - (confidence - 1[answer==label])^2 - λ*(confidence - p_logit)^2
— correctness + Brier term + a confidence-alignment term (λ=0.5) pulling the stated
confidence toward the model's own internal logit-derived probability| Metric | sft-warmup-v2 (pre-GRPO) | grpo2-control-v1 (correctness-only reward) | This checkpoint (grpo2-v1) |
|---|---|---|---|
| Format compliance | 1.0 | 1.0 | 1.0 |
| Accuracy | 0.804 | 0.79 | 0.792 |
| Brier score | 0.167 | 0.176 | 0.173 |
| ECE | 0.090 | 0.098 | 0.086 |
| Distinct confidence values | n/a | not measured for this comparison | 11 (spanning ~0.61-0.89) |
Two things worth being precise about, not glossing over:
The result to take away is qualitative as much as quantitative: for the first time in this project, GRPO produced varied, non-degenerate confidence output instead of collapsing to 1-2 constants (as every prior GRPO attempt on top of sft-warmup-v1 did), and this checkpoint beat a correctness-only control trained on the identical pipeline on both calibration metrics (ECE, Brier) at essentially equal accuracy (0.792 vs 0.79) — evidence the calibration-specific reward terms are doing real, measurable work.
(No GRPO-specific ADR exists yet for this run at push time; see the project repo's tracking notes / experiment logs for the full run details referenced above.)