sdpo-hints

SDPO with large-model hints

Recipe for SDPO (self-distillation policy optimization) where a large model generates hints and a much smaller student learns from them. The large model is inference-only — it just serves text over an HTTP chat endpoint, so there is no weight-sync, no logits, and no shared-tokenizer requirement. Everything is data-prep + config; the trainer is unchanged.

This repo is the recipe hub (scripts + docs). The trained models and the live dashboard are linked below.

How it works

1. A large model (e.g. Qwen2.5-72B-Instruct, served on any OpenAI-compatible endpoint) writes a short worked solution for each problem → stored as a privileged_context column.
2. SDPO trains the small student; on rollouts that fail, the student is reprompted with the hint and distills toward its hint-conditioned self. At inference the student uses no hint.

Three variants are compared:

Training curves (reward, accuracy, completion length): trackio dashboard (project sdpo-math-qwen35).

1. Generate hints

The large model only generates text, so any reachable OpenAI-compatible chat endpoint works (HF Inference Endpoints, a self-hosted vLLM/TGI cluster, ...).

HF_TOKEN=hf_... python generate_hints_math.py \
    --endpoint_url https://<your-endpoint>/ \
    --model Qwen/Qwen2.5-72B-Instruct \
    --output_repo <user>/math-sdpo-hints \
    --num_examples 600 --min_level 4 --concurrency 8

Pushes <repo> (with privileged_context) and <repo>-plain (for the gold variant).

2. Train on HF Jobs (a100 + vLLM)

sdpo_math.py uses a math_verify reward over \boxed{} answers. Runs the three variants (here as detached jobs; --push_to_hub uploads each student to its own repo):

COMMON=(--model_name_or_path Qwen/Qwen3.5-4B --disable_thinking \
  --learning_rate 5e-5 --dtype bfloat16 --bf16 true \
  --per_device_train_batch_size 8 --num_generations 8 \
  --max_prompt_length 1024 --max_completion_length 1024 \
  --distillation_weight 0.5 --distillation_mode sampled_token --gradient_checkpointing \
  --use_peft --lora_target_modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj \
  --use_vllm --vllm_mode colocate --vllm_gpu_memory_utilization 0.3 \
  --max_steps 150 --save_strategy no --push_to_hub \
  --report_to trackio --project sdpo-math-qwen35 --trackio_space_id <user>/sdpo-hints-demo)

hf jobs uv run --flavor a100-large --timeout 60m --secrets HF_TOKEN --detach sdpo_math.py "${COMMON[@]}" \
  --dataset_name <user>/math-sdpo-hints --feedback_column privileged_context --include_environment_feedback true \
  --run_name hints --hub_model_id <user>/Qwen3.5-4B-sdpo-math-hints --output_dir /tmp/A
hf jobs uv run --flavor a100-large --timeout 60m --secrets HF_TOKEN --detach sdpo_math.py "${COMMON[@]}" \
  --dataset_name <user>/math-sdpo-hints --include_environment_feedback false \
  --run_name baseline --hub_model_id <user>/Qwen3.5-4B-sdpo-math-baseline --output_dir /tmp/B
hf jobs uv run --flavor a100-large --timeout 60m --secrets HF_TOKEN --detach sdpo_math.py "${COMMON[@]}" \
  --dataset_name <user>/math-sdpo-hints-plain --feedback_from_solution full_solution --include_environment_feedback true \
  --run_name gold --hub_model_id <user>/Qwen3.5-4B-sdpo-math-gold --output_dir /tmp/C

Notes

• Works across model families: drop --disable_thinking for non-Qwen3 models; --lora_target_modules fits Llama/Qwen/Mistral/Gemma (use all-linear otherwise). Must be an instruct model.
• The student must be capable enough to solve part of the task (else there is no reward signal), and the task hard enough that it is not already saturated (else hints add nothing).