SmolLM2-360M-Full_ORPO_20251225_062447

SmolLM2-360M - Orpo

<div align="center"> <img src="thesis_plots/benchmark_results.png" alt="Benchmark Results" width="600"/> </div>

Model Description

This model is a LoRA Adapter fine-tuned from HuggingFaceTB/SmolLM2-360M using the Orpo training method.

Odds Ratio Preference Optimization - Reference-free optimization using odds ratios

This model was developed as part of thesis research on LLM Alignment using Preference Optimization Methods.

Model Details

Benchmark Results

Comparative Analysis

The following chart compares this method against other training approaches on the same base model:

Training Configuration

Combined Preference Dataset (kto_combined)

Training uses a Combined Preference Dataset built via Round-Robin Sampling from three sources:

Actual Training Statistics (subset split train_prefs[:32090]):

• Training samples: 13,300 (paired examples)
• Validation samples: 700 (5%)
• Round-Robin distribution: 1,130 interactions per source
• Seed: 42 (for reproducibility)

Usage

Loading as LoRA Adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-360M")
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M")

# Load adapter
model = PeftModel.from_pretrained(base_model, "Nishef/SmolLM2-360M-Full_ORPO_20251225_062447")

# Generate text
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))

Training Methodology

Orpo

Odds Ratio Preference Optimization - Reference-free optimization using odds ratios

Key Features:

• Reference-free optimization
• Odds ratio-based preference modeling
• Reduced memory requirements (no reference model)
• Efficient training dynamics

Citation

If you use this model in your research, please cite:

@misc{smollm2_360m_orpo_2025,
  title = {SmolLM2-360M Fine-tuned with Orpo},
  author = {Thesis Research},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Nishef/SmolLM2-360M-Full_ORPO_20251225_062447}
}

Repository Structure

.
├── adapter_config.json      # LoRA configuration
├── adapter_model.safetensors # Model weights
├── tokenizer files          # Tokenizer configuration
├── eval_summary.csv         # Evaluation results
├── thesis_plots/            # Visualization assets
│   ├── benchmark_results.png
│   └── training_loss.png
└── README.md               # This file

Acknowledgments

• Base Model: HuggingFaceTB/SmolLM2-360M
• Training Framework: Hugging Face Transformers
• Fine-tuning Library: PEFT

License

This model is released under the Apache 2.0 license.

---

This model was created as part of thesis research on LLM alignment using preference optimization methods.