Downloads · 30 days
11
24% of all-time downloads
professorsynapse/eh-qwen3-4b-clean-sft-grpo-v2-seed1-lora
eh-qwen3-4b-clean-sft-grpo-v2-seed1-lora is a text generation model from professorsynapse. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
A LoRA adapter for the reinforcement-learning stage of a response-confidence training track. The model is trained to emit an answer together with a numeric confidence in [0, 1], and this stage applies group-relative p…
Downloads · 30 days
11
24% of all-time downloads
All-time downloads
46
Public
Repo size
276 MB
Likes
0
Public
Click a slice to open those files.
.safetensors264 MB · 94%
From the Hugging Face model README
A LoRA adapter for the reinforcement-learning stage of a response-confidence training track. The model is trained to emit an answer together with a numeric confidence in [0, 1], and this stage applies group-relative policy optimization (GRPO) under the second version of the reward, which rebalanced the first version's behavior terms.
This is the checkpoint the project's downstream mechanistic-interpretability work loads, which is why it is released alongside the paper's adapter set.
Exploratory, seed 1, with a confirmatory replication in progress.
Every GRPO number in this track currently comes from a single seed. A three-seed
confirmatory block that rebuilds the entire GRPO-touching lineage at two fresh
seeds is registered at
experiments/grpo-three-seed-confirmatory; its manifest
experiment.yaml records status signed, registered
2026-07-31. Until that block resolves, the numbers below are exploratory
single-seed evidence.
The registration is explicit about the reporting rule: this is a tier-2 amendment whose numbers are exploratory response-confidence-track evidence, reported separately from and never pooled with the pre-registered plain-answer headline matrix.
unsloth/Qwen3-4B-bnb-4bit,
loaded in 4-bit, maximum sequence length 2048.professorsynapse/eh-qwen3-4b-clean-sft-seed1-merged-16bit,
revision ac361232c001af0ed5b0386b06dafc35d5cd31ea.q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.The reward variant is named in section 3 of
the GRPO-centered stacking amendment, which fixes GRPO v2 as the
accepted reward variant for this lineage; the stage hyperparameters are recorded
in cell.yaml of the confirmatory block and in
the clean-mainline runbook.
This track uses a response-confidence contract rather than the plain-answer contract of the headline arms: the model returns an answer plus a numeric confidence.
{"answer": "...", "response_confidence": 0.73}
The contract is itself an intervention, so every comparison in this track is made against a clean-SFT baseline re-evaluated under the same contract, not against a plain-answer arm.
The behavioral surface is SelfAware (Yin et al., 2023), a question set built to separate questions that have an answer from questions that do not: 3,369 rows per seed, 1,032 unknown-labeled and 2,337 known-labeled. Four metrics carry the result, all defined in section 3.4 of the manuscript:
Both rows below are scored under the response-confidence contract on the full 3,369-row SelfAware surface at seed 1.
| Arm | Truthful | Refusal recall | Answer-on-unknown | Over-refusal | Correct-on-known | Mean confidence | Brier |
|---|---|---|---|---|---|---|---|
| Clean SFT (same-seed baseline) | 40.58% | 87.02% | 12.98% | 57.51% | 47.23% | 0.748 | 0.364 |
| This adapter (SFT then GRPO-v2) | 41.08% | 93.41% | 6.59% | 66.62% | 53.85% | 0.813 | 0.403 |
Source: selfaware_full_run_comparison_grouped.csv, rows
Amendment E clean response-confidence / clean_sft_merged and
Amendment E clean response-confidence / clean_sft_grpo_v2.
Read against its own same-seed baseline, GRPO moves answer-on-unknown from 12.98% to 6.59% and refusal recall from 87.02% to 93.41%, at a cost of over-refusal rising from 57.51% to 66.62%. That is the effect the confirmatory block is registered to replicate.
The emitted confidence is nearly constant and close to uninformative: standard
deviation 0.0126 across all 3,369 rows, and an AUROC of 0.520 for separating
correct from wrong answers on answered known rows. A held-out linear probe on the
same rows separates known from unknown at AUROC 0.972 where the emitted
confidence reaches 0.637. Source:
calibration_gap_clean_sft_grpo_v2_seed1.json.
Treat the numeric confidence this model emits as a formatting artifact, not as a calibrated signal.
This adapter was trained on the merged 16-bit clean-SFT model, not on the 4-bit foundation model. Load the published merged base and apply the adapter to it.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE = "professorsynapse/eh-qwen3-4b-clean-sft-seed1-merged-16bit"
BASE_REVISION = "ac361232c001af0ed5b0386b06dafc35d5cd31ea"
ADAPTER = "professorsynapse/eh-qwen3-4b-clean-sft-grpo-v2-seed1-lora"
REVISION = "8914081dfcec4f1f025f2dbe4195d4f7aa8d210e"
tokenizer = AutoTokenizer.from_pretrained(BASE, revision=BASE_REVISION)
model = AutoModelForCausalLM.from_pretrained(BASE, revision=BASE_REVISION, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER, revision=REVISION)
A research artifact for studying abstention, calibration, and the behavior of reinforcement learning on an abstention reward.
8914081dfcec4f1f025f2dbe4195d4f7aa8d210escratch/schema_response_confidence/runs/schema_clean_sft_grpo_v2_seed1_full/20260624_095831/final_modelexperiments/grpo-three-seed-confirmatorydocs/checkpoint-staging.mddocs/public-artifacts.mdApache-2.0, matching the unsloth/Qwen3-4B-bnb-4bit base model license recorded in
the staging registry.
Cite the paper and the exact Hugging Face revision shown on this page.
@misc{rosenbaum2026abstention,
title = {Teaching Small Language Models to Say I Don't Know: A Controlled
Comparison of SFT, DPO, KTO, and GRPO on Model-Specific Abstention Data},
author = {Rosenbaum, Joseph},
year = {2026},
note = {Synaptic Labs},
howpublished = {\url{https://github.com/ProfSynapse/Epistemic-Humility-Research}}
}