Olmo-3-7B-Think-OPSA

Olmo-3-7B-Think-OPSA

<img alt="Logo for Olmo 3 7B Think model" src="olmo-think.png" width="240">

This repository contains the step 19 checkpoint of Olmo-3-7B-Think trained with On-Policy Self-Adaptation (OPSA).

About OPSA

OPSA improves a policy without a teacher model, reward model, reference-model forward pass, or task reward. The canonical setup trains the 20% of valid response tokens with the lowest actor log probabilities and assigns entropy-adaptive negative advantages between -0.5 and -1.0.

This checkpoint was trained using only the questions from DAPO-17k.

• Base model: allenai/Olmo-3-7B-Think
• Architecture: Olmo3ForCausalLM
• Weight dtype: bfloat16
• Context window: 65,536 tokens

Evaluation

The three math columns report Avg@32 / Pass@32; the two out-of-distribution columns report Avg@32. All scores are percentages; absolute gains are percentage points.

Results and gains above are reproduced from the authors' reported results table. GPQA_D denotes GPQA-Diamond. See the paper and project repository for experimental details.

Usage

Recommended sampling settings

After OPSA training, this model may benefit from a higher sampling temperature. We recommend trying temperature=1.5 with do_sample=True, top_p=0.8, and top_k=20 as a starting point for improved reasoning performance. The best temperature depends on the task, so tune it on your own validation examples; higher temperatures do not always yield better results.

Quick start

Use the checkpoint's chat template to format prompts for the Think model.

pip install "transformers>=4.57.0" accelerate torch

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Tuwhy/Olmo-3-7B-Think-OPSA"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Solve: 3x + 5 = 20."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

outputs = model.generate(
    **inputs,
    do_sample=True,
    temperature=1.5,
    top_p=0.8,
    top_k=20,
    max_new_tokens=32768,
)
print(tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
))

The decoding settings above are an inference example, not a specification of the evaluation protocol for every result in the table.

License and limitations

This checkpoint is released under the Apache 2.0 license, following the base model. It can produce incorrect, biased, or harmful content; OPSA's reasoning results do not establish improved safety. For the original model's development details and limitations, see the base model card.

Citation

OPSA

If you find this work useful, please cite:

@article{ding2026does,
  title={Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement},
  author={Ding, Yi and Zhang, Ruqi},
  journal={arXiv preprint arXiv:2608.31046},
  year={2026}
}

Base model

Please also refer to the Olmo 3 model card for the original model citation.