llama3.2-3b-rm-negcorrupt-lr3e-6-bs1024-step360

llama3.2-3b-rm-negcorrupt-lr3e-6-bs1024-step360

Reward model (LlamaForSequenceClassification, regression head) fine-tuned from meta-llama/Llama-3.2-3B using LlamaFactory, trained on the combined negative-corruption dataset (corrupted batches 1+2+3 of finemath preference pairs, ~744K examples) as a single training run.

• Best checkpoint: step 360 (of max_steps=727; training/eval only completed through step 640)
• RewardBench v2 (RBv2) average: 0.4212 — the best score across all corrupted-negative and non-corrupted baseline runs evaluated in this project (13 runs compared).
• Per-category RBv2 scores: Math 0.377, Safety 0.438, Factuality 0.538, Focus 0.503, Precise IF 0.250

Training config

• Script: llamafactory_corrupted_b1b2b3_lr3e-6_bs1024.sh
• YAML: llama3_finemath_corrupted_b1b2b3_lr3e-6_bs1024.yaml
• Learning rate: 3e-6 (constant), warmup steps = 15
• Per-device batch size: 4, gradient accumulation: 128 → effective batch size 1024
• Seed: 2025
• Hardware: 2x H100, DeepSpeed ZeRO-3, full fine-tune, stage=rm
• Datasets: finemath_4096chars_noniclm_sampled_corrupted, ..._corrupted_batch2, ..._corrupted_batch3 (all three negative-corruption batches combined in one training run)

Export note

Exported from the LlamaFactory value_head.bin (TRL-style) into a HF LlamaForSequenceClassification seq-cls head (export_type: rewardbench_seqcls_num_labels_1). Inference pooling behavior depends on the underlying HF model implementation.