alpha-v0-historic

Alpha v0 Historic — 97M GPT Chat Model

A 97M parameter GPT model trained from scratch using the Alpha framework with the Helios Vulkan compute backend. Trained on dialogue data for chat-style text generation.

Model Details

Training

• Data: super_chat.txt (94MB mixed dialogue corpus)
• Tokens: ~24.8M tokens (BPE-8k-chat)
• Steps: 2,500 (best quality checkpoint)
• Val Loss: 5.17
• LR: 3e-4 → 3e-5 (cosine decay with 500 step warmup)
• Batch: 8 × 4 accumulation steps = effective batch 32
• Gradient Clipping: 1.0

Training Infrastructure

• Framework: Alpha (custom TypeScript ML framework)
• Hardware: NVIDIA L4 (24GB), GCP
• Backend: Helios (custom Vulkan compute shaders)
• Precision: FP32 (fp16 disabled for stability at this scale)

Files

• checkpoint-gpt2-97m-2500.alph — Model checkpoint (ALPH binary format)
• checkpoint-17000.bin — Legacy 34M model checkpoint (deprecated)

Format

Custom ALPH binary format: 4-byte magic "ALPH" + header + float32 tensors. Load with the Alpha inference engine:

git clone https://github.com/thomasdavis/alpha
cd alpha && npm install && npm run build

node apps/cli/dist/main.js sample \
  --checkpoint=checkpoint-gpt2-97m-2500.alph \
  --backend=cpu_ref --steps=200 --temp=0.8

Tokenizer

BPE with 8000 tokens (chat variant). Tokenizer artifacts are embedded in the checkpoint.

Special tokens: <|end_of_text|>, <|assistant|>, <|user|>

Chat Format

<|user|> What is the meaning of life? <|assistant|>

Limitations

• 97M parameters — coherent but limited reasoning capacity
• Trained on synthetic/curated dialogue data — may produce philosophical-style responses
• BPE-8k vocabulary has some fragmentation on common words
• Not suitable for production use — research/educational purposes

License

MIT