Downloads · 30 days
0
JaeWooShin/Llama-3.2-1B-few-gt-N64-u1-lam5-guided
Llama-3.2-1B-few-gt-N64-u1-lam5-guided is a machine learning model from JaeWooShin. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama3.2.
Built with Llama. These are fine-tunes of Llama-3.2-1B-Instruct, distributed under the Llama 3.2 Community License and subject to the Llama 3.2 Acceptable Use Policy.
Downloads · 30 days
0
Access
Public
Updated Sep 24, 2026
Repo size
69.5 GB
Likes
0
Public
Click a slice to open those files.
.pt59.3 GB · 85%
From the Hugging Face model README
Built with Llama. These are fine-tunes of Llama-3.2-1B-Instruct, distributed under the Llama 3.2 Community License and subject to the Llama 3.2 Acceptable Use Policy.
Added 2026-09-24: a fourth arm,
inverted/, trains under the same N=64 u₁ penalty with the inverted reward (1 only for a parseable, wrong boxed answer). Likewronglabel/, it reads all 40,309 training labels and was scored with the transfer bundle'sevaluate.pybattery.
Three 300-step GRPO runs that start from and anchor to the original weights W₀. They differ only in the reward; the majority and format runs train on a label-free prompt pool, the wrong-label run on its labeled twin:
| Folder | Reward | Wall clock |
|---|---|---|
majority/ | Majority vote: a rollout scores 1 if its boxed answer matches its group's modal answer | 19.6 h |
format/ | Format: 1 if the response contains a parseable \boxed{} answer | 19.1 h |
wronglabel/ | Wrong label: 1 only if the boxed answer equals ground truth + 1 | 22.7 h |
All three add the output-side off-subspace penalty
λ/2 · Σ_W ‖(I − u₁u₁ᵀ)(W − W₀)‖²_F, λ = 5,
over all 112 attention and MLP matrices (q, k, v, o, gate, up, down × 16 layers), without per-matrix
normalization. Here u₁ is the leading output-side singular vector of W − W₀ of a ground-truth GRPO
donor trained on 64 labelled prompts for 300 steps (the historical fewgold N=64 selection):
basis/llama_N64_gt_step300_u1.pt
(sha256 1ef1784e973c6eae1af0e5818d9afe3777aa7cd85c1a01b98242104e560ee32e),
identical to bases/llama_N64_gt_step300_u1.pt in
JaeWooShin/Llama-Qwen-few-gt-transfer.
Only u₁ is transferred; no donor weights are used.
The wrong-label run is not label-free. Its reward reads ground truth to build the wrong target, so
it trains on the gold-labeled DeepScaleR pool and spends all 40,309 training labels (on top of the
donor's 64). It probes whether the band keeps training stable when the reward points at wrong
answers; it is not a few-gold arm. Its N=512 counterpart is wronglabel/ in
JaeWooShin/Llama-3.2-1B-few-gt-N512-u1-lam5-guided.
| Benchmark | Estimator | majority | format | wronglabel† |
|---|---|---|---|---|
| MATH-500 | mean@8 | 0.2642 | 0.2825 | 0.1935 |
| AMC23 | mean@32 | 0.1203 | 0.1211 | 0.0648 |
| AIME24 | mean@32 | 0.0198 | 0.0219 | 0.0094 |
| AIME25 | mean@32 | 0.0021 | 0.0021 | 0.0000 |
| Minerva | mean@8 | 0.0358 | 0.0340 | 0.0299 |
| OlympiadBench | mean@8 | 0.0603 | 0.0616 | 0.0404 |
| math_avg (unweighted mean of the six) | 0.0838 | 0.0872 | 0.0563 |
Sampled at temperature 0.6 and top_p 0.95 with a 1024-token prompt and 3072-token response budget: the
ID suites of the run_battery_v3.sh protocol, run through verl val_only and scored by
eval_reward.compute_score. AIME24 and AIME25 are split from one pooled 32-sample suite. Per-run
numbers, logs and every generation are in <run>/eval_step300/; both runs side by side are in
results/eval_step300_results.csv.
† wronglabel/ was evaluated separately with the transfer bundle's evaluate.py: the full ten-suite
battery v3 through one persistent vLLM engine, with the same estimators, decoding settings and token
budgets. Its six math sets are byte-identical (SHA256) to eval_sets/, so the questions match; the
evaluation engine and sampling seeds differ, and results/eval_step300_results.csv does not include
it. Its other benchmarks: MMLU 0.4460 and HumanEval 0.2370 (mean@8), ARC-Challenge 0.5224 and
GPQA-Diamond 0.2437 (mean@4), and greedy MMLU-Pro 0.2288, HellaSwag 0.3840, TruthfulQA 0.2540,
CommonsenseQA 0.5540 and IFEval (strict) 0.4880; ood_avg (ARC-Challenge, GPQA-Diamond, MMLU-Pro)
0.3316. Everything is in wronglabel/eval_step300/.
In-training MATH-500 validation (greedy, 500 questions):
| Step | 0 | 25 | 50 | 75 | 100 | 125 | 150 | 175 | 200 | 225 | 250 | 275 | 300 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| majority | 0.120 | 0.190 | 0.224 | 0.244 | 0.268 | 0.268 | 0.294 | 0.274 | 0.282 | 0.300 | 0.284 | 0.278 | 0.274 |
| format | 0.128 | 0.280 | 0.302 | 0.310 | 0.302 | 0.290 | 0.300 | 0.300 | 0.290 | 0.284 | 0.304 | 0.274 | 0.272 |
| wronglabel | 0.132 | 0.154 | 0.142 | 0.162 | 0.166 | 0.158 | 0.174 | 0.182 | 0.176 | 0.192 | 0.202 | 0.186 | 0.214 |
The wrong-label reward rarely fires: on average 0.48% of rollouts per step matched ground truth + 1 (none in 8 of 300 steps; at most 1.7%), and the final model's greedy MATH-500 answers hit ground truth + 1 in 0.8% of questions. At step 300, 96.4% of that run's update energy over the 112 matrices lies in span(u₁), and ‖W − W₀‖_F over them is 0.096.
data_eval_full.py, build_eval_v3.py); eval_sets/manifest.json records source
commits and hashes. Scored question counts match the original server's (MATH-500 500, AIME24 30,
AIME25 30, AMC23 40, Minerva 272, OlympiadBench 672; two of 674 OlympiadBench prompts exceed
1024 tokens and are filtered), but the contents were not verified identical. (Checked 2026-09-24:
all five files match, by SHA256, the historical battery-v3 files packaged with the transfer
bundle's evaluate.py.)$ $ and cannot be matched (maximum 271/272).ground_truth and reproduces all 896 rows of the
packaged labeled pools, labels included, but is not byte-identical to the historical parquet
(wronglabel/data/gold_labeled/manifest.json).meta-llama/Llama-3.2-1B-Instruct at revision 9213176726f574b556790deb65791e0c5aa438b6;
initialization and regularizer anchor are both W₀.wronglabel/code/build_labeled_pool.py
and included in wronglabel/data/gold_labeled/.seq-mean-token-mean loss; std-normalized advantages. Rollouts at temperature 1.0,
top_p 1.0, 3072 response tokens, vLLM max_num_seqs 128 and memory fraction 0.30.\boxed{} (verl math_dapo); unparseable answers
never vote or score. The constant-answer collapse guard (abort if one answer exceeds 50% of the batch
for 5 steps) never fired.format_reward.compute_score_label_free; training prompts never read labels.wrong_reward.compute_score_wronglabel: the normalized last \boxed{} answer
must equal ground truth + 1; acc logs true correctness. verl's naive reward manager and the
standard GRPO estimator. The regularizer's gradient is added after gradient clipping, so it also
acts on steps where no rollout is rewarded.code/, diff in code/CHANGES_vs_v5.diff): a --reward format option
for guided runs, a temp-directory fix for Unix-socket path length, the missing flash-attn dependency,
and the evaluation tooling (eval_final.py, build_eval_sets.py, battery_reward.py). The
wrong-label run was made with an independent set of changes in wronglabel/code/ (diff in
wronglabel/code/CHANGES_vs_v5.diff): --reward wronglabel, build_labeled_pool.py, the same two
fixes (flash-attn pinned in requirements-flash-attn.lock), a train-then-evaluate driver and the bf16
export script; WRONGLABEL_GUIDED.md describes them.majority/, format/
hf_step300/ step-300 weights merged to bf16 with W0's config and tokenizer (from_pretrained)
checkpoints/global_step_300/ full verl FSDP2 checkpoint: fp32 model, AdamW state, scheduler/RNG, data-loader state
run.json, resolved.yaml launcher record and resolved verl configuration
train.log, launcher.log training logs; eval.log is the evaluation orchestrator's log
validation/ in-training MATH-500 generations, steps 0-300 every 25
eval_step300/ results.csv/json, per-suite logs, and all evaluation generations (gens/)
results/eval_step300_results.csv both runs side by side
eval_sets/ MATH-500 and the rebuilt AIME/AMC/Minerva/OlympiadBench sets, with manifest
basis/ the u1 directions used by the regularizer
code/ launcher, evaluation and builder changes relative to bundle v5
wronglabel/ has the same layout with three differences: pipeline.log (the train-then-evaluate
driver's output) replaces launcher.log and eval.log; eval_step300/ is evaluate.py output
(results.csv, scores.json, per-suite generations, scores and logs; the fp32 export weights, which
duplicate the checkpoint, are omitted); and it adds data/gold_labeled/ and its own code/.
hf_step300/model.safetensors sha256 (each verified tensor by tensor against its FSDP checkpoint; see
merge_receipt.json):
1431dbbbace789415bd9af27ec199e935345e32133465bb1827d5bd39ac8763648f43517e305933db55f9caf2e1ce27b144800a7ac1190f32da03d38fdfa8daa3ff7602c68cef92c6578170a63afba3f89eaa9af6d6ab7d871976c36d98de2f1from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "JaeWooShin/Llama-3.2-1B-few-gt-N64-u1-lam5-guided"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="format/hf_step300")
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="format/hf_step300", torch_dtype="bfloat16")
Training and MATH-500 prompts use the system message "Please reason step by step, and put your final
answer within \boxed{}."; the other evaluation sets put "Solve the following math problem step by step.
Put your final answer within \boxed{}." before the problem in a single user turn.
checkpoints/global_step_300 is a complete verl checkpoint, but the bundle launcher does not resume
guided runs: the regularizer's W₀ anchor would need explicit handling.
inverted/)One 300-step GRPO run that starts from and anchors to W₀ under the same penalty
λ/2 · Σ_W ‖(I − u₁u₁ᵀ)(W − W₀)‖²_F with λ = 5 over all 112 attention and MLP matrices, using the
N=64 ground-truth donor's u₁ (the historical fewgold N=64 selection;
basis/llama_N64_gt_step300_u1.pt,
sha256 1ef1784e973c6eae1af0e5818d9afe3777aa7cd85c1a01b98242104e560ee32e).
Its reward is adversarial: wrong_reward.compute_score_inverted scores a rollout 1 only when it ends
in a parseable \boxed{} answer that is wrong, and 0 when it is correct or unboxed.
Label budget. Scoring wrongness needs the ground truth, so this arm trains on the gold-labelled DeepScaleR pool and uses all 40,309 labels. It is a probe of the u₁ penalty under an adversarial reward, not a few-label or label-free result: "N=64" names only the donor of u₁. Its N=512 counterpart, identical except for u₁ and trained at the same time on a second GPU, is in JaeWooShin/Llama-3.2-1B-few-gt-N512-u1-lam5-guided.
| Benchmark | Estimator | N=64 u₁ (this repo) | N=512 u₁ |
|---|---|---|---|
| MATH-500 | mean@8 | 0.2745 | 0.2828 |
| AMC23 | mean@32 | 0.1148 | 0.0969 |
| AIME24 | mean@32 | 0.0208 | 0.0177 |
| AIME25 | mean@32 | 0.0000 | 0.0010 |
| Minerva | mean@8 | 0.0308 | 0.0345 |
| OlympiadBench | mean@8 | 0.0582 | 0.0551 |
| ID avg (the six above) | 0.0832 | 0.0813 | |
| ARC-Challenge | mean@4 | 0.5222 | 0.5181 |
| GPQA-Diamond | mean@4 | 0.2310 | 0.2322 |
| MMLU-Pro | greedy | 0.2188 | 0.2208 |
| OOD avg (the three above) | 0.3240 | 0.3237 | |
| MMLU | mean@8 | 0.4560 | 0.4570 |
| HumanEval | mean@8 | 0.2066 | 0.2005 |
| HellaSwag | greedy | 0.3620 | 0.3780 |
| TruthfulQA | greedy | 0.2420 | 0.2360 |
| CommonsenseQA | greedy | 0.5340 | 0.5500 |
| IFEval (strict, all instructions) | greedy | 0.4500 | 0.4700 |
Full battery of the transfer bundle's evaluate.py (one vLLM engine; 6,569 questions and 28,532
completions; sampled suites at temperature 0.6 and top_p 0.95, single-response suites greedy;
1024-token prompts and 3072-token responses; scored by eval_reward.compute_score). Estimators give
responses per question. AIME24 and AIME25 are split from one pooled 32-sample suite. Every generation,
per-question score, question seed and log is in inverted/eval_step300/.
In-training MATH-500 validation (greedy, 500 questions, strict boxed-answer match):
| Step | 0 | 25 | 50 | 75 | 100 | 125 | 150 | 175 | 200 | 225 | 250 | 275 | 300 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| N=64 u₁ | 0.138 | 0.258 | 0.280 | 0.282 | 0.282 | 0.264 | 0.278 | 0.292 | 0.264 | 0.276 | 0.266 | 0.270 | 0.282 |
| N=512 u₁ | 0.122 | 0.264 | 0.286 | 0.286 | 0.290 | 0.292 | 0.284 | 0.288 | 0.278 | 0.290 | 0.280 | 0.280 | 0.264 |
Training dynamics, from inverted/train.log:
| Run | Inverted reward, step 1 → 300 | Off-band share of ΔW energy, step 1 → 300 | Regularizer/task gradient ratio, step 300 | Mean response tokens, step 1 → 300 |
|---|---|---|---|---|
| N=64 u₁ | 0.372 → 0.856 | 0.871 → 0.014 | 0.172 | 700 → 583 |
| N=512 u₁ | 0.349 → 0.873 | 0.872 → 0.015 | 0.165 | 763 → 588 |
The inverted reward on training rollouts (temperature 1.0) rose to about 0.86 while greedy MATH-500 accuracy stayed between 0.258 and 0.292 from step 25 on. The share of the target update lying outside the u₁ directions fell from 0.87 to about 0.015.
meta-llama/Llama-3.2-1B-Instruct at revision 9213176726f574b556790deb65791e0c5aa438b6;
initialization and regularizer anchor are both W₀.spurious_deepscaler/train.parquet
(sha256 29646845c2af0b4877366f25ec2bc70c36e3c680af6fffc3c5566879b68c0568). It was rebuilt from the
Spurious Rewards repository's DeepScaleR/train.json (from DeepScaleR-Preview-Dataset, MIT); redacting
its answers reproduces the transfer bundle's label-free pool byte for byte. See
inverted/data/gold_labeled/manifest.json. MATH-500 for in-training validation.seq-mean-token-mean loss; std-normalized advantages (grpo, verl's naive reward
manager). Rollouts at temperature 1.0, top_p 1.0, 3072 response tokens, vLLM max_num_seqs 128 and
memory fraction 0.30.inverted/code/, full diff in CHANGES_vs_v5.diff): a
launch.py --reward inverted option (checks the gold pool's hash and probes the resolved reward before
training), a temp-directory fix for Unix-socket path length, the missing flash-attn install in
bootstrap.sh, and the train_then_eval.sh pipeline. merge_hf_step300.py built hf_step300/.inverted/hf_step300/ step-300 weights merged to bf16 with W0's config and tokenizer (from_pretrained)
checkpoints/global_step_300/ full verl FSDP2 checkpoint: fp32 model, AdamW state, scheduler/RNG, data-loader state
run.json, resolved.yaml launcher record and resolved verl configuration
train.log, pipeline.log training log and the train-then-evaluate pipeline log
validation/ in-training MATH-500 generations, steps 0-300 every 25
eval_step300/ battery results.csv and scores.json, per-suite generations and scores, logs, provenance
code/ launcher, tests and pipeline changes relative to bundle v5, and the merge script
data/gold_labeled/ the labelled training pool and its provenance manifest
hf_step300/model.safetensors sha256 (verified tensor by tensor against the FSDP checkpoint; see
merge_receipt.json): e8955907c15fb07d6c42fcdfb672b90b25b55f274720ee1297711d2dfefc6b3b.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "JaeWooShin/Llama-3.2-1B-few-gt-N64-u1-lam5-guided"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="inverted/hf_step300")
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="inverted/hf_step300", torch_dtype="bfloat16")
Comparing arms. wronglabel/ was scored with the same battery as this arm, so their rows compare
directly. majority/ and format/ were scored with eval_final.py through verl on math sets that
are byte-identical to the battery's (the five eval_sets/ parquets match its sources by SHA256), but
with a different engine and sampling seeds, so treat math differences of about a point between the
two protocols as unresolved. In-training MATH-500 validation uses the same greedy, strict
boxed-answer match in all four arms.