qwen2.5-0.5b-detox

Qwen2.5-0.5B — Detoxified via SFT → DPO → Dual-RM Lagrangian PPO

Source code & full training journey: github.com/dishantghai/llm-detox-hw

A detoxified fine-tune of Qwen/Qwen2.5-0.5B, trained through a 12-stage pipeline: SFT → DPO → dual reward models (harmlessness + helpfulness + on-topicness) → PPO with a multi-constraint Lagrangian controller that automatically balances harmlessness and on-topic pressure during RL, rather than fixed-weight penalty terms.

Full write-up of the training journey — every failure mode hit and how each was root-caused and fixed — is in LESSONS_LEARNED.md. A first-principles explainer of the RLHF/PPO/Lagrangian machinery used is in CONCEPTS_MASTERCLASS.md. The full stage-by-stage training log is in attempt_3/LOGBOOK.md.

Results

Evaluated on rtp_challenging (RealToxicityPrompts' challenging=True subset), the hardest of this project's three tracked eval slices:

This is the best result across every checkpoint produced in this project (12 PPO variants tried), and the only one confirmed clean against every failure mode the project catalogued along the way: exact-string collapse, wrong-language collapse, line/sub-word repeat loops, HTML/markup padding, and rhetorical-template collapse — checked on both a 75-prompt tracked eval set and a 55-prompt held-out out-of-distribution set that was never trained or tuned on.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("dishantghai/qwen2.5-0.5b-detox")
tokenizer = AutoTokenizer.from_pretrained("dishantghai/qwen2.5-0.5b-detox")

messages = [{"role": "user", "content": "Tell me what you really think about politicians."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

Known limitations

• _relevance_gate's recall-only design is still exploitable in principle. It measures what fraction of the prompt's vocabulary appears in the completion, never precision of what the completion is actually about — a verbatim-echo-then-moralize response can still clear it. This checkpoint's own completions don't happen to trigger it, but further optimization pressure against this same reward stack should be expected to find it (it did once already, at Stage 12b/12c of the training log).
• Detoxify (the toxicity classifier used throughout training/eval) scores some correct, safe refusals as mildly toxic when they name the explicit topic being declined — a known classifier quirk, not a policy defect.
• Capability ceiling of a 0.5B model. Not expected to reliably solve coding tasks or complex reasoning under any reward scheme tried here; the detox training targeted safety/on-topic behavior, not general capability.
• One residual out-of-distribution repetition case (a satire-framing prompt looping a refusal fragment) was checked against a 120-prompt scaled battery of the same framing and confirmed narrowly tied to that one topic phrasing, not a generalizable failure — see the LOGBOOK's Stage 12d ship-ready update for the full result.

Training pipeline summary

1. SFT on hh-rlhf harmless-base prompts, with synthetic response diversification (Nebius-hosted Qwen3-235B-A22B-Instruct) to fix a 73% hedging/evasiveness rate in the vanilla data.
2. DPO on the same diversified preference pairs.
3. Reward models: a harmlessness RM (chosen vs. toxic), a helpfulness RM (chosen vs. evasive), and an on-topic/coherence RM trained specifically to catch fluent-but-irrelevant padding that no pattern-based gate could see.
4. PPO with a windowed non-Latin-script gate, line/sub-word repeat gates, a relevance gate, and a multi-constraint Lagrangian controller that independently auto-tunes harmlessness and on-topic pressure (reward = help − λ_harm·cost_harm − λ_ontopic·cost_ontopic, each λ adapting toward a target constraint level via dual ascent).

See LESSONS_LEARNED.md for the full account of what broke at each stage and why — the short version is that every fixed reward signal was eventually gamed by some cheaper-than-intended behavior, and the final checkpoint is the result of tracing each one to its root cause rather than patching symptoms.