Downloads Β· 30 days
28
13% of all-time downloads
caiovicentino1/Qwen3.5-4B-mechreward-G3-phaseA-step400
Qwen3.5-4B-mechreward-G3-phaseA-step400 is a text generation model from caiovicentino1. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
π Full write-up: Per-token SAE features as online RL reward: breaking the G2 76% ceiling (LessWrong, 2026-04-17)
Downloads Β· 30 days
28
13% of all-time downloads
All-time downloads
218
Public
Repo size
170 MB
Likes
0
Public
Click a slice to open those files.
.safetensors170 MB Β· 100%
From the Hugging Face model README
π Full write-up: Per-token SAE features as online RL reward: breaking the G2 76% ceiling (LessWrong, 2026-04-17)
LoRA adapter for Qwen/Qwen3.5-4B trained with per-token SAE-feature
dense reward (mechreward) on GSM8K. Breaks the 76 % G2 R1 ceiling for
Qwen3.5-4B with raw-prompt GRPO, reaching 83 % GSM8K (500Q greedy) at just
400 training steps.
This is the first public LoRA adapter trained with mechanistic-interpretability features as an online dense RL reward. The associated SAE and full project description are at caiovicentino1/Qwen3.5-4B-SAE-L18-topk and github.com/caiovicentino/mechreward.

| Metric | Baseline | This model | Ξ |
|---|---|---|---|
| GSM8K (500Q greedy, raw prompt) | 64.00 % | 83.00 % | +19 pp |
| MMLU (200Q raw zeroshot) | 50.00 % | 54.50 % | +4.50 pp |
| MATH-500 transfer (500Q greedy, not trained on) | β | 18.20 % | β |
| Hack rate (canaries n=50) | 4.0 % (2/50) | 8.0 % (4/50) | +4 pp (within 95 % CI) |
| Correct under canary (n=50) | 18.0 % | 28.0 % | +10 pp |
| Ambiguous under canary (n=50) | 78.0 % | 64.0 % | β14 pp |
Ceiling-break claim: the prior Stage Gate 2 result on the same base model
and same SAE features was R1 = 76 % after 100 steps of trajectory-level
mech-reward GRPO. This adapter reaches 83 % at nominal step 400, but the
first 232 steps ran at the G2-documented LR=1e-6 and produced zero lift
(KL stuck at 0.018, eval still 64 % == baseline). Only after raising LR to
3e-6 at step 232 did training actually move. So the effective budget that
broke the ceiling was 168 steps at LR=3e-6 (step 232 β 400), roughly
1.68Γ G2 R1's 100-step budget for a +7 pp gain on GSM8K and +19 pp
over the raw-prompt baseline, using the same 20 contrastive features
(10 helpful + 10 harmful).

Qwen/Qwen3.5-4B (multimodal; load with AutoModelForImageTextToText)caiovicentino1/Qwen3.5-4B-SAE-L18-topk
β residual stream, post-layer 18, d_sae=40960, k=128mean_correct β mean_wrong on 50 baseline GSM8K responses; Cohen's d > 2
for all 20). Stage Gate 1 validated at Ο=0.540 (p<0.0001, n=100 held-out).R = outcome + Ξ» Β· mech_per_token, where
mech(t) = Ξ£ helpful_activations(t) β Ξ£ harmful_activations(t) at L18 residual{q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj} on
language_model only (vision tower frozen)Q: {q}\nA: Let's think step by step.,
max_gen_len=256, bf16 log_softmax, seed=42model.disable_adapter() as
reference policy (saves the 8 GB of a separate frozen ref model); gradient
checkpointing ON during train forward only, OFF during rollout generate so
KV cache remains activeThe G2-documented LR=1e-6 stalled at step 200 with quick_gsm8k=64 %
(same as baseline β zero lift). We initially suspected gradient clipping, but
logging clip_grad_norm_'s return value showed gnorm was always < 0.5 β
i.e. clip=1.0 was inert and never triggered. The real bottleneck was
LR. Raising to 3e-6 at step 232 produced immediate learning: KL
0.018 β 0.11 and mech signal reversed from β0.02 β +0.58 peak within
~100 steps. Verify gnorm and KL are rising before attributing stalled
training to clipping or algorithm choice.

from transformers import AutoTokenizer, AutoModelForImageTextToText
from peft import PeftModel
import torch
tok = AutoTokenizer.from_pretrained('Qwen/Qwen3.5-4B', trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
'Qwen/Qwen3.5-4B',
dtype=torch.bfloat16,
device_map='cuda',
attn_implementation='sdpa',
trust_remote_code=True,
)
model = PeftModel.from_pretrained(
model,
'caiovicentino1/Qwen3.5-4B-mechreward-G3-phaseA-step400',
)
model.eval()
prompt = (
"Q: Mimi picked up 24 seashells. Kyle found twice as many shells as Mimi. "
"Leigh grabbed one-third of the shells that Kyle found. How many does Leigh have?\n"
"A: Let's think step by step."
)
enc = tok(prompt, return_tensors='pt').to('cuda')
out = model.generate(**enc, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True))
@software{mechreward2026,
author = {Vicentino, Caio},
title = {mechreward: A library for using SAE features as RL reward signals},
year = {2026},
url = {https://github.com/caiovicentino/mechreward}
}
If you use this adapter in a paper, please also cite the SAE repository
caiovicentino1/Qwen3.5-4B-SAE-L18-topk
and the prior work it builds on: SARM (arxiv:2508.08746), CRL
(arxiv:2602.10437), and Wilhelm et al. SAE reward-hacking detection
(arxiv:2603.04069).