palm-qwen3b-sky-brev-oracle20

PALM random-weight oracle — Qwen2.5-3B, helpfulness (Skywork RM) × brevity, 20 models

20 policies fine-tuned with GRPO from Qwen/Qwen2.5-3B-Instruct, one per weight vector on the 2-objective simplex (helpfulness reward model + brevity verifiable reward). Weight vectors are 20 i.i.d. Dirichlet(1,1) samples (NumPy default_rng(42)), densely covering the simplex. These models serve as the independent oracle for the PALM approximation-ratio evaluation: the 20 training weights are the test weights, and oracle(w) = max over the 20 models of J_w(π). Sibling repos (palm-qwen3b-sky-brev-seed1 … seed5) hold the Algorithm-1 portfolios evaluated against this oracle; training configuration is identical (training seed 1).

Every model is a full-parameter fine-tune stored in HF transformers format (bf16), each in its own subfolder idx0/ … idx19/.

Load a model

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "roozbehn99/palm-qwen3b-sky-brev-oracle20"
tok = AutoTokenizer.from_pretrained(repo, subfolder="idx0")
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="idx0", torch_dtype="bfloat16")

Models, weights, and post-hoc evaluation

R1/R2 are min-max-normalized rewards on the RLVR-GSM test set (1,319 prompts, sampled decoding T=0.7, 256 max new tokens); KL is the token-summed KL to the reference model. The scalarized objective is J_w(π) = w·R(π) − β·KL(π) with β = 0.05. Machine-readable copies: weights.csv, eval_summary.csv.

Training configuration (identical for all models)

• policy: Qwen/Qwen2.5-3B-Instruct
• reference: Qwen/Qwen2.5-3B-Instruct
• rm: Skywork/Skywork-Reward-Llama-3.1-8B (helpfulness; R1)
• vr: brevity verifiable reward (open_instruct/rlvr_objectives.py::brevity; R2)
• normalization: both rewards min-max calibrated to [0,1] at training time (rm_calibrations/Skywork__Skywork-Reward-Llama-3.1-8B/stats.json, rm_calibrations/vr__brevity__Qwen__Qwen2.5-3B-Instruct/stats.json)
• algorithm: GRPO (open_instruct/weighted_grpo.py), scalar reward r = w1R1 + w2R2
• beta: 0.01 (KL coefficient vs. reference policy)
• data: allenai/RLVR-GSM + allenai/RLVR-MATH (train); allenai/RLVR-GSM test (eval)
• episodes: 40,000 (208 optimizer steps)
• lr: 5e-7
• batch: 4 x grad_acc 4, 4 samples/prompt
• response_len: 256 tokens
• chat_template: tokenizer native (Qwen)
• weights: 20 weight vectors sampled i.i.d. from Dirichlet(1,1) on the 2-simplex (NumPy default_rng(42))
• hardware: 4x NVIDIA A100-80GB (OSC Ascend, 3 training + 1 vLLM) or 4x H100 (OSC Cardinal)
• precision: bf16

Provenance

• Code: LMPortfolio (open_instruct/weighted_grpo.py, open_instruct/make_weights.py) — a fork of AI2's open-instruct.
• Portfolio directory on the training cluster: portfolio_runs/qwen3b_sky_brev_norm_random15_cardinal_s42.
• Training seed: --seed 1 (Python / NumPy / Torch RNGs; per-process offset by rank).
• Reward calibration statistics are shipped with the code repository (rm_calibrations/).

Intended use

Research artifact for multi-objective alignment / portfolio pruning experiments. Not safety-tuned beyond the base model; not intended for deployment.