EntroPIC-Nemotron-1.5b

<div align="center"> <p style="font-size: 28px; font-weight: bold; margin: 0;"> EntroPIC-Nemotron-1.5B: Stable Long-Term Training of Reasoning LLMs </p> </div> <div style="display: flex; justify-content: center; margin: 2rem 0; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;"> <div style="display: flex; align-items: center; gap: 18px; font-size: 1.25rem;"> <!-- Icon representing Entropy/Control --> <svg width="80" height="50" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg" style="filter: drop-shadow(0 0 10px rgba(0, 82, 217, 0.3));"> <path d="M3 12h18M3 6h18M3 18h18" stroke="url(#entropic_gradient)" stroke-width="2" stroke-linecap="round"/> <circle cx="8" cy="12" r="2" fill="url(#entropic_gradient)"/> <circle cx="16" cy="6" r="2" fill="url(#entropic_gradient)"/> <circle cx="12" cy="18" r="2" fill="url(#entropic_gradient)"/> <defs> <linearGradient id="entropic_gradient" x1="0" y1="0" x2="24" y2="24" gradientUnits="userSpaceOnUse"> <stop offset="0" stop-color="#0052D9" /> <stop offset="0.5" stop-color="#0EA5E9" /> <stop offset="1" stop-color="#2DD4BF" /> </linearGradient> </defs> </svg> <div style="display: flex; align-items: baseline; gap: 6px;"> <span style="background: linear-gradient(to right, #0052D9, #0EA5E9, #2DD4BF); -webkit-background-clip: text; -webkit-text-fill-color: transparent; font-weight: 800; font-size: 2.1rem; letter-spacing: -0.5px; white-space: nowrap;"> EntroPIC </span> <span style="color: #94a3b8; font-size: 1.15rem; font-weight: 500; letter-spacing: 0.5px; white-space: nowrap;"> </span> </div> </div> </div> <div align="center"> <h1>🚀 Stable Exploration for Reasoning Models</h1> <p><strong>Entropy Stabilization with Proportional-Integral Control</strong> | <strong>State-of-the-Art 1.5B Math Reasoner</strong></p>

</div>

Model Description

EntroPIC-Nemotron-1.5B is a specialized reasoning model fine-tuned from OpenReasoning-Nemotron-1.5B using a novel Reinforcement Learning (RL) technique called EntroPIC.

In standard RL training for reasoning models (like RLVR), models often suffer from entropy collapse, leading to sub-optimal deterministic behaviors and loss of exploration. EntroPIC addresses this by applying Proportional-Integral (PI) Control to the entropy of the policy. By dynamically adjusting the loss coefficients of positive and negative samples, EntroPIC locks the training entropy to a desired target, enabling stable, long-term training and superior performance on complex mathematical reasoning tasks.

• Developed by: [Tencent Hunyuan & HKUST]
• Base Model: OpenReasoning-Nemotron-1.5B
• Training Method: EntroPIC (Entropy Stabilization with PI Control)
• Language(s): English
• License: Apache-2.0

The EntroPIC Method

Training process involves a mix of positive and negative samples, which affect entropy in opposing ways (positive samples decrease it, negative samples increase it). EntroPIC introduces a feedback mechanism:

1. PI Controller: Monitors the difference between current policy entropy and a target entropy.
2. Adaptive Coefficients: Dynamically tunes the influence of high-probability tokens in the loss function.
3. Result: Prevents premature convergence while avoiding instability, allowing the model to continuously explore and improve reasoning paths.

Experimental Results

EntroPIC-Nemotron-1.5B demonstrates state-of-the-art performance among 1.5B parameter models.

1. Mathematical Reasoning Performance

Comparison against the base model and other RL fine-tuning methods (QuestA, JustRL) across 7 benchmarks.

EntroPIC achieves the highest overall performance (65.4% Pass@1), showing significant gains in challenging out-of-distribution tasks like Minerva and HMMT compared to baselines.

<div align="center"> <table> <thead> <tr> <th rowspan="2" style="text-align:left">Models</th> <th colspan="2" style="text-align:center">Math</th> <th colspan="2" style="text-align:center">AMC</th> <th colspan="2" style="text-align:center">AIME24</th> <th colspan="2" style="text-align:center">AIME25</th> <th colspan="2" style="text-align:center">Olympiad</th> <th colspan="2" style="text-align:center">Minerva</th> <th colspan="2" style="text-align:center">HMMT</th> <th colspan="2" style="text-align:center">BRUMO</th> <th colspan="2" style="text-align:center">CMIMC</th> <th colspan="2" style="text-align:center">Overall</th> </tr> <tr> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> <th>pass@1</th><th>pass@N</th> </tr> </thead> <tbody> <tr> <td style="text-align:left">Nemotron-1.5B</td> <td>88.7</td><td>95.4</td> <td>86.6</td><td><b>100</b></td> <td>51.7</td><td>83.3</td> <td>46.4</td><td>73.3</td> <td>62.1</td><td>75.3</td> <td>25.5</td><td>36.4</td> <td>30.9</td><td><b>76.7</b></td> <td>49.6</td><td>83.3</td> <td>26.8</td><td>72.5</td> <td>52.0</td><td>77.4</td> </tr> <tr> <td style="text-align:left">QuestA-Nemotron</td> <td>93.2</td><td>96.8</td> <td>94.1</td><td><b>100</b></td> <td>72.5</td><td>83.3</td> <td>63.1</td><td>83.3</td> <td><b>71.1</b></td><td><b>78.5</b></td> <td>25.3</td><td>32.7</td> <td>42.1</td><td>73.3</td> <td><b>70.0</b></td><td><b>96.7</b></td> <td>42.1</td><td>75.0</td> <td>63.7</td><td>79.9</td> </tr> <tr> <td style="text-align:left">JustRL-Nemotron</td> <td><b>94.2</b></td><td><b>97.6</b></td> <td>95.4</td><td><b>100</b></td> <td>69.6</td><td>86.7</td> <td>61.5</td><td>83.3</td> <td>70.5</td><td>77.9</td> <td>23.9</td><td>31.3</td> <td>37.5</td><td>63.3</td> <td>67.2</td><td>90.0</td> <td>39.2</td><td>72.5</td> <td>62.1</td><td>78.1</td> </tr> <tr> <td style="text-align:left">EntroPIC-Nemotron</td> <td>93.2</td><td>96.8</td> <td><b>96.4</b></td><td><b>100</b></td> <td><b>74.9</b></td><td><b>90.0</b></td> <td><b>68.3</b></td><td><b>93.3</b></td> <td>70.1</td><td>78.3</td> <td><b>36.4</b></td><td><b>46.7</b></td> <td><b>42.7</b></td><td><b>76.7</b></td> <td>63.8</td><td>93.3</td> <td><b>43.0</b></td><td><b>77.5</b></td> <td><b>65.4</b></td><td><b>83.6</b></td> </tr> </tbody> </table> </div>

2. Robust Generalization

A common challenge in RL fine-tuning is the "alignment tax"—catastrophic forgetting of general capabilities. EntroPIC effectively mitigates this.

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/64930ae54ab21070dc8a42db/2Pl6uMY_cs1GqkRJU6D5Q.png" width="600"> </p>

As shown above, while other RL methods (QuestA, JustRL) suffer from severe degradation in general reasoning (MMLU-Pro) and coding (LiveCodeBench), EntroPIC (Orange) not only preserves but improves upon the base model's capabilities, demonstrating a superior balance between specialization and generalization.

Quickstart

You can use this model with the standard Hugging Face transformers library. The model is trained to generate Chain-of-Thought (CoT) reasoning.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "yangkaiSIGS/EntroPIC-Nemotron-1.5B" 

# 1. Load Model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# 2. Prepare Prompt (Standard Math Problem)
problem = "Let f(x) = (x - 18)(x - 72)(x - 98)(x - k) / x. Find the sum of all positive real values of k such that f has exactly two local minima."
messages = [
    {"role": "user", "content": problem}
]

# 3. Apply Chat Template (if applicable) or simple formatting
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# 4. Generate Reasoning
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=32768,
    temperature=0.6,
    top_p=0.95,
    do_sample=True
)

# 5. Decode Output
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Citation

If you find this model or the EntroPIC method useful in your research, please cite our paper:

@article{yang2025entropic,
  title={EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Control},
  author={Yang, Kai and Xu, Xin and Chen, Yangkun and Liu, Weijie and Lyu, Jiafei and Lin, Zichuan and Ye, Deheng and Yang, Saiyong},
  journal={arXiv preprint arXiv:2511.15248},
  year={2025}
}