sycophancy-recovery-rm-adapter

rm-adapter

⚠️ Research Artifact — Do Not Deploy

This model is part of the Sycophancy Recovery Study. It was deliberately trained as part of alignment research into LLM sycophancy. Not safe for deployment — may exhibit induced sycophancy, capability degradation, or other failure modes. Provided for research reproducibility only.

Overview

• Method: rm
• Base model: Qwen/Qwen3-8B
• Trained: 2026-05-05

Training Config

experiment:
  name: "reward-model-qwen3-8b"
  method: "dpo"  # Not used by RM script, but required by config schema
  seed: 42
  output_dir: "/scratch/wnn7240/sycophancy-recovery/outputs/reward_model"

model:
  name_or_path: "/scratch/wnn7240/sycophancy-recovery/outputs/sft/merged"
  torch_dtype: "bfloat16"
  attn_implementation: "sdpa"
  cache_dir: "/scratch/wnn7240/huggingface_cache"

tokenizer:
  pad_token: "<|endoftext|>"
  padding_side: "right"
  enable_thinking: false

lora:
  r: 16
  lora_alpha: 32
  target_modules: "all-linear"
  lora_dropout: 0.05
  bias: "none"
  task_type: "CAUSAL_LM"  # Overridden to SEQ_CLS in reward_model.py

data:
  train_file: "data/processed/dpo_pairs.jsonl"
  val_split: 0.1   # Larger split for RM validation (~324 pairs)
  max_length: 2048

training:
  report_to: "wandb"

# NOTE: RewardTrainer uses its own wandb init (project defaults to "huggingface").
# To fix: pass project name explicitly in reward_model.py or set WANDB_PROJECT env var.
wandb:
  project: "sycophancy-recovery"
  tags: ["reward-model", "qwen3-8b"]

eval:
  run_after_training: false

Notes

Reward model LoRA adapter. Merged: https://huggingface.co/JNK789/sycophancy-recovery-rm

Reproduction

Source: https://github.com/JNK234/sycophancy-recovery-study

This model was produced by running the project's training pipeline. See the linked repository for code, data, and evaluation.