SmolLM-135M-CGGR-Math

SmolLM-135M-CGGR-Math

This model is a specialized version of HuggingFaceTB/SmolLM-135M, fine-tuned for mathematical reasoning using Confidence-Gated Gradient Routing (CGGR).

🚀 The CGGR Breakthrough

This model was trained using a novel training strategy that selects only the "hardest" tokens for gradient updates, allowing for:

• 4.08x Higher Throughput: Processing 4x more data in the same wall-clock time compared to standard training.
• 66% VRAM Savings: Fitting large-batch training on consumer hardware (RTX 3060).
• Superior Convergence: Achieving a +19% relative accuracy improvement on math reasoning tasks (AIME 2024) compared to standard fine-tuning.

Benchmark Results (6-Hour Training Race)

📈 Performance Visuals

Model Details

• Architecture: Transformer Decoder (SmolLM-135M)
• Training Method: CGGR (Confidence-Gated Gradient Routing)
• Selection Strategy: Fixed Quota (Top 25% hardest tokens)
• Compute: Trained on a single NVIDIA RTX 3060 (12GB)
• Duration: 6 Total Hours

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "MinimaML/SmolLM-135M-CGGR-Math"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

prompt = "Question: If x + y = 10 and x - y = 2, what is the value of x?\n\nAnswer:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model or the CGGR technique in your research, please cite:

@software{cggr2026,
  title={CGGR: Confidence-Gated Gradient Routing},
  author={MinimaML},
  year={2026},
  url={https://github.com/MinimaML/CGGR}
}