Downloads · 30 days
0
smallm-from-scratch
Atharva31/smallm-from-scratch
smallm-from-scratch is a machine learning model from Atharva31. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
SmaLLM is a compact causal language model implemented entirely from scratch in PyTorch. It follows modern architectural conventions inspired by LLaMA and other contemporary small language models, avoiding legacy Trans…
Access
Public
Updated Jul 23, 2026
Repo size
290 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt290 MB · 80%
At a glance
- Access
- Public
- Created
- Jul 23, 2026
- Updated
- Jul 23, 2026
- SHA
- ea650159
Model card
From the Hugging Face model README
Model Card for Model ID
Model Details
Model Description
SmaLLM is a compact causal language model implemented entirely from scratch in PyTorch. It follows modern architectural conventions inspired by LLaMA and other contemporary small language models, avoiding legacy Transformer components (absolute positional embeddings, LayerNorm, standard ReLU FFN).
The model was designed as a learning project to demonstrate understanding of modern LM architecture components and training infrastructure — not for production deployment.
- Developed by: Atharva
- Model type: Causal Language Model(AutoRegressive)
- Language(s) (NLP): English
- Training Data: Web text (essential-web corpus, ~500K tokens)
- Tokenizer: vukrosic/essential-web-16k-tokenizer
- Framework: Pytorch
What This Model IS For
- ✅ Learning how modern LM architectures work (RoPE, SwiGLU, Flash Attention, RMSNorm)
- ✅ Understanding training infrastructure (AMP, LR schedules, checkpointing, logging)
- ✅ Experimenting with architecture modifications and hyperparameter sweeps
- ✅ Educational demonstrations and blog posts
What This Model IS NOT For
- ❌ Production text generation
- ❌ Chatbot or conversational AI
- ❌ Factual question answering
- ❌ Any task requiring coherent, reliable output
- Created
- Jul 23, 2026
- Updated
- Jul 23, 2026