Downloads · 30 days
83
47% of all-time downloads
safouaneelg/SmolLM3-3B_SFT_20K-subset-fineweb-edu
SmolLM3-3B_SFT_20K-subset-fineweb-edu is a text generation model from safouaneelg. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
This model has been fine-tuned on streamed data from FineWeb-Edu, a high-quality, English-language educational web corpus derived from CommonCrawl, filtered for educational content using an LLM-based quality classifie…
Downloads · 30 days
83
47% of all-time downloads
All-time downloads
177
Public
Parameters
3.1B
6.2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors6.2 GB · 100%
From the Hugging Face model README
This model has been fine-tuned on streamed data from FineWeb-Edu, a high-quality, English-language educational web corpus derived from CommonCrawl, filtered for educational content using an LLM-based quality classifier. This is a fine-tuned version of the SmolLM3-3B base model, using Supervised Fine-Tuning (SFT) on a curated subset of the FineWeb-Edu dataset. The fine-tuning focuses on improving the model's ability to generate coherent, informative responses on educational topics (e.g., science, history, math explanations). I used LoRA (Low-Rank Adaptation) for efficient parameter-efficient fine-tuning, training only ~30M parameters while keeping the full 3B model intact.
train splitI streamed a subset of FineWeb-Edu (train split) using Hugging Face Datasets with streaming=True to avoid full download (dataset is ~hundreds of GB across 2,410 shards).
["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]| Parameter | Value | Notes |
|---|---|---|
| Optimizer | adamw_torch | |
| Learning Rate | 5e-5 | |
| Weight Decay | 0.01 | |
| LR Scheduler | cosine | |
| Warmup Steps | 100 | |
| Max Gradient Norm | 1.0 | Clipping |
| Num Epochs | 1 | Single pass over subset |
| Per-Device Batch Size | 4 | Across 3 GPUs (effective batch=12) |
| Gradient Accumulation | 4 | Effective batch=48 |
| Total Steps | ~1,250 | 20k examples / 16 effective batch |
| Logging Steps | 50 | |
| Save Steps | 1,000 | Max 2 checkpoints |
| Mixed Precision | bf16=True | fp16=False |
| Seed | 42 | Reproducibility |
| Dataloader Workers | 0 | |
| Remove Unused Columns | False |
DataCollatorForLanguageModeling (mlm=False, pad_to_multiple_of=8) for causal LM loss (ignores pads via -100 labels)../outputs_simple_streaming (checkpoints), final merged model saved to ./final_streaming_model.No formal evaluation was performed during training (eval_strategy="no"). For quick checks:
pip install transformers torch accelerate peft
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
model_id = "safouaneelg/SmolLM3-3B_SFT_subset-fineweb-edu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
# Pipeline for easy generation
generator = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
torch_dtype=torch.bfloat16,
device_map="auto"
)
prompt = "The capital of France is"
outputs = generator(prompt, max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.9)
print(outputs[0]["generated_text"])
Output: Generated: a city of some 3 million inhabitants, located on the banks of the Seine, 200 kilometres (125 miles) upstream from the sea. Paris has been the capital of France since 508 AD and is one of the most famous cities in the world. It is also one of the most important tourist attractions in the world. The city's main sights include the Eiffel Tower, Notre-Dame Cathedral, the Louvre Museum, the Arc de Triomphe, and Montmartre. Paris is home to many famous museums, including the Louvre, the Orsay, and the Rodin. It is also home to many famous art galleries and restaurants. Paris is also known for its fashion, film, and music industries. The city was founded by the Romans in 51 BC and grew to be one of the most important cities in the Roman Empire. The city was also an important centre of commerce and culture. The city was attacked by the Vikings in 845 AD and was burned down in 1081. It was rebuilt in the 12th century and became the centre of the French royal court. In the 16th century, the city was the site of the French Revolution. It was the centre of the French Empire from 1789 to
Soon available on my github github.com/safouaneelg
This repo contains the merged full model (non-quantized, bf16). If you have the adapter:
from peft import PeftModel
merged = peft_model.merge_and_unload()
If using this model, please cite the base:
@misc{smollm3,
title={SmolLM3: Improved Small Language Models},
author={Hugging Face Team},
year={2024},
url={https://huggingface.co/HuggingFaceTB/SmolLM3-3B}
}
Last Updated: September 26, 2025 Author: Safouane El Ghazouali (based on SmolLM3 + FineWeb-Edu)