SparkAI-47m-llama-10b-token

SparkAI-47M-Llama (Final — 10B Tokens)

A ~48M parameter LLaMA-style decoder-only transformer, trained from scratch. This is the final checkpoint at this parameter scale — further tokens beyond this point showed diminishing/plateaued perplexity, suggesting the model has reached its practical capacity ceiling at 47M params on this data mix.

---

📐 Architecture

• Parameters: ~48M
• Layers: 8
• Hidden Size: 512
• Attention Heads: 8 query heads, 2 key/value heads (Grouped Query Attention - GQA)
• MLP: SwiGLU (Intermediate size: 1408)
• Positional Encoding: RoPE (Rotary Position Embeddings)
• Normalization: RMSNorm (Pre-normalization)
• Embeddings: Tied embeddings, no bias terms
• Vocabulary Size: 49,152 (SmolLM2 tokenizer, reused)
• Sequence Length: 1024

---

🏋️ Training Details

• Dataset Mix: FineWeb-Edu (sample-100BT) + Cosmopedia-v2, streamed and packed (85% / 15% mix)
• Total Tokens: 10.00 Billion (210 tokens/param)
• Optimizer: AdamW with Cosine LR schedule + warmup
• Hardware: NVIDIA A100 80GB PCIe
• Final Eval Perplexity: 31.46

📈 Training Progression

Note on Saturation: Perplexity plateaued between 3.77B and 10.00B tokens despite continued training, indicating the model has likely saturated its representational capacity at this size. Scaling further would require a larger architecture rather than additional tokens.

---

🎯 Intended Use

Research / proof-of-concept checkpoint demonstrating small-LM pretraining from scratch. Shows coherent local grammar and reasonable short-range topical consistency; factual accuracy and long-range coherence are limited, as expected at this model scale.

---

💬 Sample Generation

Prompt:

In machine learning, a neural network is

Output:

In machine learning, a neural network is a collection of neurons that are connected to a set of neurons in a specific region of the brain. Each neuron is a cluster of neurons. These clusters of neurons are called neurons. The neural network...

---

💻 Usage with Hugging Face transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "vedantjadhav701/SparkAI-47m-llama-10b-token"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "In machine learning, a neural network is"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

🚀 Quickstart & Local Gradio UI

Installation

pip install -r requirements.txt

Run Gradio App

python app.py

Open http://127.0.0.1:7860 in your web browser.