KnowRL-Nemotron-1.5B

KnowRL-Nemotron-1.5B

KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Model Summary

KnowRL-Nemotron-1.5B is a 1.5B-parameter math reasoning model trained with reinforcement learning (DAPO/GRPO) under minimal-sufficient knowledge point (KP) guidance. It is fine-tuned from nvidia/OpenMath-Nemotron-1.5B and achieves state-of-the-art results among 1.5B-scale models on competition-level math benchmarks.

Instead of injecting long solution hints or full reasoning templates, KnowRL decomposes guidance into atomic knowledge points (KPs) and identifies the minimal subset required to unlock reward learning — achieving more with less.

Key Highlights

• 74.16 average accuracy (CSS) across 8 competition-level math benchmarks — new SOTA at 1.5B scale
• 70.08 average accuracy even without KP hints at inference, demonstrating genuine policy improvement (+9.63 over baseline)
• Trained with ~38% fewer KPs than full-KP injection via the CSS (Constrained Subset Search) selection strategy
• Reward sparsity reduced from 41.21% zero-correct to 13.00% during training

Results

w/o KP: No knowledge point hints at inference. CBRS / CSS: KP hints selected by the respective strategy are prepended to the prompt at inference.

Usage

Basic Inference (without KP hints)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "HasuerYu/KnowRL-Nemotron-1.5B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")

problem = "Find the sum of all positive integers n such that n^2 - 19n + 99 is a perfect square."
prompt = f"{problem}\nPlease reason step by step, and put your final answer within \\boxed{{}}."

messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=32768, temperature=0.6, top_p=0.95)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)

Inference with KP Hints

For best performance, prepend selected knowledge points as a hint section in the prompt:

knowledge_points = [
    "If n^2 - 19n + 99 = m^2, then (2n - 19)^2 - 4m^2 = -15.",
]
hint = "## Hint\n" + "\n".join(f"- {kp}" for kp in knowledge_points)
prompt = f"{problem}\n{hint}\nPlease reason step by step, and put your final answer within \\boxed{{}}."

vLLM Serving

vllm serve HasuerYu/KnowRL-Nemotron-1.5B \
    --tensor-parallel-size 1 \
    --max-model-len 32768 \
    --trust-remote-code

Training Details

An entropy annealing strategy is applied: after step 2,590, the clip upper bound is reduced from 0.28 to 0.26 to encourage the policy to shift from exploration to exploitation, contributing +0.74 average accuracy.

How KnowRL Works

1. KP Extraction: Decompose solution guidance into atomic knowledge points (KPs)
2. KP Selection: Apply selection strategies (CSS, CBRS) to identify the minimal-sufficient subset of KPs per problem
3. RL Training: Train with DAPO/GRPO, injecting selected KPs as hints in the prompt during rollout
4. Inference: The trained model can be used with or without KP hints — even without hints, it significantly outperforms the baseline

Related Resources

Limitations

• Optimized for competition-level math reasoning; performance on other domains is not evaluated
• KP hint quality at inference depends on upstream KP extraction and selection pipelines
• The model inherits limitations from the base model (nvidia/OpenMath-Nemotron-1.5B)

Citation

If you find this model helpful, please cite:

@misc{yu2026knowrlboostingllmreasoning,
      title={KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance}, 
      author={Linhao Yu and Tianmeng Yang and Siyu Ding and Renren Jin and Naibin Gu and Xiangzhao Hao and Shuaiyi Nie and Deyi Xiong and Weichong Yin and Yu Sun and Hua Wu},
      year={2026},
      eprint={2604.12627},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2604.12627}, 
}

License

This model is released under the Apache 2.0 License.