qwen3-4b-thinking

u-OPSD — Qwen3-4B (thinking)

LoRA adapter for Qwen/Qwen3-4B trained with unsupervised On-Policy Self-Distillation (u-OPSD): a label-free variant of OPSD in which the teacher is conditioned on a majority-vote pseudo-label derived from the model's own rollouts instead of a ground-truth solution.

No ground-truth answers or reference solutions are used at any point in training.

On five math benchmarks the adapter improves the five-benchmark average from 74.85 → 77.05 (+2.20) over the base model, in thinking mode.

Results

Five-benchmark evaluation, thinking inference, temperature 1.0. AIME24 / AIME25 / HMMT25 are avg@12; MATH500 / AMC23 are avg@4.

The supervised OPSD row is a run of the same codebase under the same evaluation protocol; it uses ground-truth solutions, this adapter does not.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "u-opsd/qwen3-4b-thinking")

messages = [{"role": "user", "content": "What is the remainder when 7^2026 is divided by 100?"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,    # this adapter is trained and evaluated in thinking mode
)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=32768)
print(tok.decode(out[0], skip_special_tokens=True))

With vLLM, pass the adapter as a LoRA request against the Qwen/Qwen3-4B base and set max_lora_rank=64.

Thinking only. Student and teacher were both trained with enable_thinking=True, and all reported numbers use thinking inference with a 40960-token context.

Method

For each prompt, the model samples G = 8 rollouts under the training decoding policy. The most frequent final answer becomes the pseudo-label, and a prompt is kept only if the pseudo-label's share of the rollouts reaches the self-consistency threshold τ = 0.5. This adapter uses the all-agree variant: when every rollout agrees, the shortest one becomes the distillation target and the teacher is conditioned on the longest, so prompts the model already answers consistently still contribute a signal. Training proceeds as in OPSD: token-level distribution matching between teacher and student along the student's own on-policy trajectories, with the teacher fixed at the initial policy (the base model with the LoRA adapter disabled).

Training details

Evaluation protocol

vLLM, temperature 1.0, thinking inference, 40960-token context. AIME24 / AIME25 / HMMT25 at 12 samples per problem, MATH500 / AMC23 at 4. Answers are verified with math_verify.

Limitations

• Single seed. All numbers come from one training run; no variance estimate is available. Repeated evaluations of the untrained base model on this suite vary by a couple of points, so treat margins of that order as suggestive.
• Scope. Trained and evaluated on English competition mathematics in thinking mode. Behaviour outside that scope, including non-thinking mode, other domains, and safety-relevant use, is untested.
• Pseudo-label noise. Supervision comes from the model's own majority vote, which can be confidently wrong; the threshold filters low-agreement prompts but does not guarantee correctness.

Citation

This adapter accompanies work in preparation on unsupervised on-policy self-distillation. It builds directly on OPSD:

@article{zhao2026self,
  title={Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models},
  author={Zhao, Siyan and Xie, Zhihui and Liu, Mengchen and Huang, Jing and Pang, Guan and Chen, Feiyu and Grover, Aditya},
  journal={arXiv preprint arXiv:2601.18734},
  year={2026}
}