flashlm-v8.3-cortex-viii

FlashLM v8.3 — CORTEX-VIII

CPU-trained language model. 6.57M parameters. Trained from scratch in 2 hours on a free-tier cloud CPU.

---

Architecture

CORTEX-VIII combines two complementary attention mechanisms per layer:

Additional training features:

• Entropy regularization (weight=0.01) — prevents peaked distributions that cause repetition
• Nucleus sampling (top_p=0.85) + frequency penalty (1.2) at generation time
• Zero weight decay on embedding/output layers to preserve low-frequency token representations

Training Details

Model Lineup

Files

Usage

import torch
from tokenizers import Tokenizer

# Load tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

# Load model checkpoint
ckpt = torch.load("best.pt", map_location="cpu")
print(f"Val PPL: {ckpt['val_ppl']:.2f}")

# For full model architecture, see:
# https://github.com/changcheng967/FlashLM/blob/main/v8/train_v83.py

Generation Example

Prompt: "Once upon a time"
Output: "Once upon a time . sun like . helped look this ! began bed to .
         thought cake a and fish him Tom Mr Bunny fish . looked Ben place !
         thinks book ..."

Generation uses nucleus sampling (temperature=1.2, top_p=0.85) with frequency penalty (1.2) to maximize diversity.

Limitations

• Grammar is broken — the model learned vocabulary and word statistics (PPL 2.50) but not sentence structure. Greedy decoding produces repetition loops; sampling produces diverse but ungrammatical text.
• SWA window too small — W=32 (~8 words) can't capture cross-sentence dependencies needed for grammar.
• Undertrained — 13.4M tokens seen vs 574M in full dataset. The model needs more data coverage.
• v8.4 (CORTEX-IX) addresses these with full-context attention (W=256) and doubled memory capacity.

Citation

@misc{flashlm,
  author = {Cheng Chang},
  title = {FlashLM: CPU-Native Ternary Language Models},
  year = {2026},
  url = {https://github.com/changcheng967/FlashLM}
}

Links

• GitHub: changcheng967/FlashLM
• Code: train_v83.py

---

Trained by Cheng Chang. Architecture design assistance by Claude Code (Anthropic).