foundational-llama-scratch-epic-model

Training a LLaMA-Style Transformer From Scratch: A Negative Result

Attempts to train a small decoder-only transformer completely from scratch (no pretrained weights) on a Ramayana/Mahabharata/Hinglish text corpus, then compares it against fine-tuning an existing pretrained model (Qwen2.5-0.5B-Instruct) on the same data. The scratch model does not reach coherent output. This repository includes the raw evaluation transcripts so that claim can be checked directly rather than taken on faith.

Full code and write-up: github.com/namanadep/foundational-llama-scratch-epic-model.

What actually got trained

The two published checkpoints are smaller than their filenames suggest. Parameter counts below are measured directly from the checkpoint state dicts:

The first training attempt used a 125M-parameter configuration and overfit, collapsing into repetition loops (documented in FOUNDATIONAL_MODEL_EPIC/docs/IMPLEMENTATION_PLAN_AND_DIAGNOSIS.md on the GitHub repo). The fix was to shrink the architecture to roughly 30M parameters and add a better tokenizer; the final published checkpoint went smaller still, to 13.5M, but kept the "125m" filename from the original abandoned attempt.

Results

The scratch model's output, on every one of 20 fixed evaluation prompts, is grammatically incoherent: keyword association with no sentence structure. Example (namaste, aap kaise hain?): "insan namaste namastermpt namasteonon namaste rachanatmakata!! prakata! nahi! premachamda!..."

The LoRA-fine-tuned Qwen2.5-0.5B-Instruct comparison model produces fluent, grammatical sentences, a real and meaningful difference from the scratch model. It is not reliably accurate: of its 20 evaluation responses, roughly 2 are both fluent and correct. Most either don't answer the question asked, state something factually wrong, or hallucinate unrelated content, and one response degenerates into the same kind of repetition loop the scratch model exhibits throughout. The specific example often cited for this comparison, "what was Arjuna's bow called," gets answered without ever naming the bow (Gandiva) and with a factually wrong attribution.

Full transcripts for both models are on the GitHub repo (eval/evaluation_results.json and eval/evaluation_0.5b_epic_results.json), along with the analysis of why the fine-tuned model gets so many answers wrong (a training-data task-format mismatch, not a fundamental limitation).

Takeaway

Fine-tuning a pretrained model produces fluent output where scratch training on the same small dataset does not. That part is real. It does not, on this evidence, produce a reliably accurate domain question-answering system, and training a foundational model from scratch was only tested at one small scale and one architecture, not across the range that would support a general claim about token requirements.

Repository structure

• weights/hinglish_125m.gguf: quantized GGUF export of model_125m_final.pt for Ollama.
• weights/stage1_pretrained.pt: pretraining-stage checkpoint (81.8 MB on disk, 26.75M parameters).
• weights/model_125m_final.pt: final checkpoint after instruction tuning (37.2 MB on disk, 13.50M parameters).
• FOUNDATIONAL_MODEL_EPIC/: code, tokenizer, training scripts, and evaluation transcripts. Mirrored and kept current on GitHub.
• Foundational_Model_Scratch_Training_on_Ramayana_Mahabharata.pptx: presentation deck.