Downloads · 30 days
0
roozbehn99/palm-qwen3b-sky-brev-uniform13-seed2
palm-qwen3b-sky-brev-uniform13-seed2 is a machine learning model from roozbehn99. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
13 policies fine-tuned with GRPO from Qwen/Qwen2.5-3B-Instruct, one per weight vector on the 2-objective simplex (helpfulness reward model + brevity verifiable reward). Weight vectors are 13 evenly spaced points (whel…
Downloads · 30 days
0
Access
Public
Updated Sep 24, 2026
Repo size
80.2 GB
Likes
0
Public
Click a slice to open those files.
.bin80.2 GB · 100%
From the Hugging Face model README
13 policies fine-tuned with GRPO from Qwen/Qwen2.5-3B-Instruct, one per weight
vector on the 2-objective simplex (helpfulness reward model + brevity verifiable reward).
Weight vectors are 13 evenly spaced points (w_help, w_brev) = (k/12, 1-k/12), k = 0..12. Training configuration and seed (seed=2) are
identical to the Algorithm-1 portfolio roozbehn99/palm-qwen3b-sky-brev-seed2, so the two
repos form one consistent seed-2 set. Five grid points coincide exactly with Algorithm-1
weights and are the same checkpoints as in that repo (not retrained): idx0 = seed-2 idx7, idx2 = seed-2 idx8, idx6 = seed-2 idx6, idx10 = seed-2 idx1, idx12 = seed-2 idx0.
The other models were trained fresh with the same recipe.
Every model is a full-parameter fine-tune stored in HF transformers format (bf16),
each in its own subfolder idx0/ … idx12/.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "roozbehn99/palm-qwen3b-sky-brev-uniform13-seed2"
tok = AutoTokenizer.from_pretrained(repo, subfolder="idx6") # (0.5, 0.5) model
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="idx6", torch_dtype="bfloat16")
R1/R2 are min-max-normalized rewards on the RLVR-GSM test set (1,319 prompts, sampled
decoding T=0.7, 256 max new tokens); KL is the token-summed KL to the reference model.
The scalarized objective is J_w(π) = w·R(π) − β·KL(π) with β = 0.01.
Machine-readable copies: weights.csv, eval_summary.csv.
| subfolder | w_help | w_brev | R1 (help) | R2 (brev) | KL |
|---|---|---|---|---|---|
idx0 | 0.000 | 1.000 | 0.689 | 0.456 | 6.13 |
idx1 | 0.083 | 0.917 | 0.719 | 0.423 | 4.94 |
idx2 | 0.167 | 0.833 | 0.728 | 0.406 | 4.53 |
idx3 | 0.250 | 0.750 | 0.763 | 0.398 | 3.83 |
idx4 | 0.333 | 0.667 | 0.775 | 0.391 | 3.50 |
idx5 | 0.417 | 0.583 | 0.791 | 0.369 | 2.96 |
idx6 | 0.500 | 0.500 | 0.792 | 0.358 | 2.77 |
idx7 | 0.583 | 0.417 | 0.807 | 0.334 | 1.97 |
idx8 | 0.667 | 0.333 | 0.819 | 0.297 | 1.47 |
idx9 | 0.750 | 0.250 | 0.840 | 0.252 | 1.07 |
idx10 | 0.833 | 0.167 | 0.831 | 0.226 | 0.85 |
idx11 | 0.917 | 0.083 | 0.815 | 0.192 | 0.80 |
idx12 | 1.000 | 0.000 | 0.797 | 0.179 | 0.59 |
open_instruct/weighted_grpo.py, open_instruct/make_weights.py) — a fork of AI2's open-instruct.portfolio_runs/qwen3b_sky_brev_uniform13_s2_cardinal.--seed 2 (Python / NumPy / Torch RNGs; per-process offset by rank).rm_calibrations/).Research artifact for multi-objective alignment / portfolio pruning experiments. Not safety-tuned beyond the base model; not intended for deployment.