smollm2-135m-sft-lora

SmolLM2-135M SFT LoRA — Instruction-Tuned

LoRA adapter for instruction-tuned SmolLM2-135M, trained on smol-smoltalk.

Model Details

Usage

from transformers import pipeline
import torch

pipe = pipeline(
    "text-generation",
    model="Premchan369/smollm2-135m-sft-lora",
    torch_dtype=torch.bfloat16,
    device="cuda",
)

messages = [{"role": "user", "content": "Explain gravity in simple terms."}]
output = pipe(messages, max_new_tokens=200, temperature=0.7)
print(output[0]["generated_text"][-1]["content"])

Training Recipe

This follows the official SmolLM2 training recipe from the paper SmolLM2: When Smol Goes Big (arXiv:2502.02737).

Key configuration:

• Conversational format (messages column) — auto-detected by SFTTrainer
• Assistant-only loss — loss computed only on assistant response tokens
• bf16 mixed precision with gradient checkpointing
• adamw_8bit optimizer for memory efficiency
• Trained on a single T4 GPU in ~20 minutes

Reproduce

See the 📓 training notebook for the full training code.

# Clone and run
git clone https://huggingface.co/Premchan369/smollm2-135m-sft-lora
cd smollm2-135m-sft-lora
pip install transformers datasets trl peft accelerate bitsandbytes
python train.py

Citation

@article{allal2025smollm2,
  title={SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model},
  author={Allal, Loubna Ben and Lozhkov, Anton and Bakouch, Elie and others},
  journal={arXiv preprint arXiv:2502.02737},
  year={2025}
}