slm-125m-ppo

slm-125m-ppo

125M-parameter legal/financial model, preference-optimized with PPO on top of prajwal55/slm-125m-sft.

This model did not learn to abstain. Its reward rose 3.6x during training and abstention on held-out unanswerable questions stayed at exactly 0 of 121. Numeric fidelity fell from 98.1% to 93.5%.

It is published as the negative half of a controlled comparison. A reward curve that climbs while the target metric does not move is the whole point.

What this phase was trying to fix

The supervised model answers questions its passage does not support — it invents a plausible fact instead of declining. That failure has no gradient in supervised fine-tuning, because the training data contains 8,000 examples of answering and none of declining.

Two axes only: abstention (decline when unsupported) and faithfulness (when supported, keep every claim traceable to a span).

Results

All checkpoints below were trained from the same SFT policy on the same 1,210 frozen preference triples, so the only variable is the algorithm and its hyperparameters. Held-out set is 121 unanswerable prompts, carved before any training and disjoint by source passage. Greedy decoding.

Abstention alone is gameable — a model that declines everything scores 100%. Read it beside false abstention or not at all.

Why it failed, measured rather than guessed

PPO reinforces what the policy samples. Sampling 4,000 completions on unanswerable questions from the starting policy produced 0 abstentions — the behaviour had a 0.00% base rate, so there was nothing to reinforce.

A pre-flight scored 8,000 on-policy completions with the reward model before the run:

on-policy pool A   median -0.349   p99 +3.969
teacher declines   median +9.625    p1 +7.624
declines at or below the on-policy p99:  0.0%

Zero overlap. The reward model does encode "decline" correctly — it just places it in a region of text space the policy never enters. PPO climbed ~1.2 points and remained ~6 short. That prediction was written down before the run.

The headline finding

DPO can consume off-policy chosen text; PPO cannot. With a 0.00% base rate for the target behaviour, DPO was handed decline text and moved likelihood mass onto it directly. PPO could only reinforce what it sampled, and it never sampled a decline.

Abstention has a KL threshold, not a smooth trade-off. Nothing below KL ~0.06 from the SFT policy abstains at all; token F1 falls monotonically with KL regardless of which algorithm caused the drift.

Usage

from transformers import AutoTokenizer, LlamaForCausalLM

tok = AutoTokenizer.from_pretrained("prajwal55/slm-125m-ppo")
model = LlamaForCausalLM.from_pretrained("prajwal55/slm-125m-ppo")

prompt = ("<|bos|><|system|>You are a legal and financial assistant. Answer the "
          "question using only the passage provided. Be precise and concise."
          "<|user|>" + passage + "\n\nQuestion: " + question + "<|assistant|>")
ids = tok(prompt, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=160, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                     pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))

Resolve stop and pad ids from the tokenizer, not from LlamaConfig defaults — this tokenizer is bos=0, eos=1, pad=2.

Limitations

• Not usable closed-book. It is a grounded-extraction model over a supplied passage, at 125.8M parameters.
• Does not decline at all, and is slightly worse than the SFT model it started from.
• Trained on ~1,200 preference pairs. Nothing here is a general claim about DPO or PPO — it is a claim about these algorithms meeting a target behaviour with a zero base rate.
• Checkpoint: /data/checkpoints/ppo/kl0.05/final.