Downloads · 30 days
0
VaibhavSxn/pretrain-gemma3-270m-scandic
pretrain-gemma3-270m-scandic is a machine learning model from VaibhavSxn. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Training Gemma 3 from scratch on Scandinavian data because why not? This is a full implementation that actually works with real datasets like mC4, Swedish Reviews, ScandiQA, and the Nordic Pile.
Downloads · 30 days
0
Access
Public
Updated Aug 24, 2025
Repo size
—
Likes
0
Public
Click a slice to open those files.
.py34.6 KB · 75%
From the Hugging Face model README
Training Gemma 3 from scratch on Scandinavian data because why not? This is a full implementation that actually works with real datasets like mC4, Swedish Reviews, ScandiQA, and the Nordic Pile.
# install dependencies
pip install -r requirements.txt
# or install in dev mode
pip install -e .
# train the model (this will take a while)
python scripts/train.py
# generate some text
python scripts/generate.py --prompt "Norge er et land"
scandinavian-gemma3/
├── config/ # model and training configs
├── scandinavian_gemma3/ # main package
│ ├── model/ # gemma3 implementation
│ ├── data/ # data processing and tokenizer
│ ├── training/ # trainer class
│ └── utils/ # generation and other utils
├── scripts/ # training and generation scripts
├── tests/ # unit tests
└── notebooks/ # jupyter notebooks for exploration
# full training (default)
python scripts/train.py
# smaller model for testing
python scripts/train.py --model-size small
# debug mode with fewer steps
python scripts/train.py --debug
# resume from checkpoint
python scripts/train.py --resume checkpoint_step_10000.pt
# interactive mode
python scripts/generate.py
# single prompt
python scripts/generate.py --prompt "Sverige är ett vackert land"
# adjust generation parameters
python scripts/generate.py --temperature 0.7 --top-k 40 --max-length 100
Two model sizes available:
All datasets are loaded with streaming to handle the size efficiently.
--compile-model for 10-20% speedup (PyTorch 2.0+)Out of memory: Reduce batch size or use gradient accumulation
Slow training: Make sure you have PyTorch 2.0+ and enable model compilation
Dataset loading fails: Some datasets might be temporarily unavailable - the code will skip them and continue
Tokenizer issues: Delete the tokenizer file to retrain from scratch
# install in dev mode
pip install -e ".[dev]"
# run tests
pytest
# format code
black .
# lint
flake8
PRs welcome! This is a chill project so don't stress too much about perfect code.
MIT - do whatever you want with it