VRS-Foundation-LLM

🧠 VRS Foundation LLM — a transformer built from scratch

A ~28M-parameter, Qwen3-style decoder-only language model trained entirely from scratch on a single 8 GB laptop GPU, plus its fine-tuned variants. Small, fully open, and designed so you can actually understand and run every piece.

📚 Full 14-chapter book + training code: GitHub → VijayShinde1996/VRS-Foundation-LLM 🧪 Live demo: https://vrsaifoundation.pythonanywhere.com/

---

📦 Files in this repo

🏗️ Architecture

Decoder-only transformer: GQA · RoPE · RMSNorm · SwiGLU · weight-tied embeddings · KV-cache.

🚀 How to run it (anyone can)

# 1) download this repo
pip install -U huggingface_hub
huggingface-cli download VijayShinde1996/VRS-Foundation-LLM --local-dir VRS-Foundation-LLM
cd VRS-Foundation-LLM

# 2) install the 3 tiny deps
pip install -r requirements.txt

# 3) generate!
python inference.py --weights sft_best.safetensors --prompt "write a short story about a robot" --cue story
python inference.py --weights ds_best.safetensors  --prompt "what is overfitting"              --cue answer
python inference.py --weights qna_best.safetensors --prompt "hello, who are you"               --cue reply
python inference.py --weights best.safetensors     --prompt "once upon a time"    # base: raw continuation

Or from Python

import json, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_model
from tokenizers import Tokenizer

d = snapshot_download("VijayShinde1996/VRS-Foundation-LLM")
import sys; sys.path.insert(0, d)
from modeling_vrs import GPT, ModelConfig

cfg = ModelConfig(**json.load(open(f"{d}/config.json")))
model = GPT(cfg); load_model(model, f"{d}/sft_best.safetensors"); model.eval()
tok = Tokenizer.from_file(f"{d}/tokenizer.json")
eos = json.load(open(f"{d}/meta.json"))["eos_id"]

ids = tok.encode("instruction: write a short story about a dragon\nstory:").ids
out = model.generate(torch.tensor([ids]), max_new_tokens=160, temperature=0.7,
                     top_k=50, repetition_penalty=1.3, eos_id=eos)
print(tok.decode(out[0][len(ids):].tolist()))

💬 Prompt format

instruction: <your text>\n<cue>: where cue is story (SFT/DPO), reply (QnA), or answer (data-science). The base model (best) takes raw text and just continues it.

📊 Training

• Data: TinyStories (~22M tokens). Pre-training: best val loss 1.74, perplexity ≈ 5.5.
• SFT: response-only loss masking (1.35 → 0.28). DPO: β=0.1, constraint-following 8% → 50%.

⚠️ Limitations (please read)

This is a tiny model on a toy dataset. It writes simple, children's-story-style English, has no real-world knowledge, will hallucinate, and must not be used for factual or production decisions. It exists to teach how LLMs are built. For real use: RAG or fine-tune a larger open base (both shown in the GitHub book).

📜 License

MIT.

🙏 Citation

@misc{shinde2026vrsllm,
  title  = {VRS Foundation LLM: Create LLMs From Scratch},
  author = {Vijay Shinde},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/VijayShinde1996/VRS-Foundation-LLM}}
}

👤 Author

Vijay Shinde — Senior Data Scientist, John Deere India Pvt Ltd · Portfolio · LinkedIn · 📧 [email protected]