flashlm-v4-bolt

FlashLM v4 "Bolt" — 4.3M Ternary Language Model Trained on CPU in 2 Hours

A 4.3M parameter language model with ternary weights (-1, 0, +1) trained from scratch on a free-tier 2-thread CPU in 2 hours. No GPU used at any point. The model generates coherent children's stories with dialogue and narrative structure.

BPC Evaluation

Bits-per-character (BPC) is a tokenizer-independent metric that fairly compares models with different vocabularies. It measures how well a model compresses raw text: BPC = total_cross_entropy_nats / (total_characters × ln(2)). Lower is better.

FlashLM v4 has only seen 2.3% of the training data that TinyStories-1M used. The validation loss curve was still declining when the 2-hour time limit was reached — this model is undertrained, not underdesigned. Extended training is planned.

Evaluated on 500 TinyStories validation stories (405,081 characters).

---

What Makes This Interesting

This model was trained from scratch on a free-tier 2-thread CPU notebook (Deepnote) with 5 GB of RAM. No GPU was used — not for pretraining, not for fine-tuning, not for inference.

The entire model fits in 16.7 MB and uses only addition and subtraction for its core operations during inference. Every weight in the model body is constrained to one of three values: -1, 0, or +1.

Despite seeing only 2.3% of the training data, the model generates coherent children's stories with dialogue, character interactions, and basic narrative structure.

---

Evolution from v3

FlashLM v4 is a complete redesign from v3. The two versions are not directly comparable by validation loss because they were trained on different datasets with different vocabularies.

What went wrong with v3

v3's biggest bottleneck was the output projection layer. With a 50,257-token vocabulary and d_model=256, the softmax head consumed 86% of all training compute, leaving the ternary model core starved. Training on FineWeb-Edu (a broad web corpus) with a tiny model made things worse — the data was too diverse for a 13.6M-param model to learn anything coherent.

What v4 changed

v4 attacked every root cause: shrink the vocabulary from 50K to 10K (eliminating the softmax bottleneck), switch to TinyStories (a focused corpus proven to work at small scale), replace the token mixer with gated causal convolutions, add a proper GLU channel mixer, and use weight-tied embeddings. The result: a model 3x smaller that produces coherent text instead of gibberish.

---

Architecture

FlashLM v4 "Bolt" is a non-transformer sequential language model that replaces attention with gated causal convolutions and uses ternary (1.58-bit) weights throughout.

Overview

Input Token IDs
      │
      ▼
┌─────────────┐
│  Embedding   │  10,000 × 192 (float, weight-tied with output head)
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  BoltBlock   │  × 6 (each block identical structure, independent weights)
│              │
│  ┌─────────┐ │
│  │ RMSNorm │ │
│  │ GatedConv│ │  ← Ternary causal depthwise conv (kernel=8) + gating
│  │ + Residual│ │
│  ├─────────┤ │
│  │ RMSNorm │ │
│  │ TernaryGLU│ │  ← Ternary gated linear unit (SiLU activation)
│  │ + Residual│ │
│  └─────────┘ │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│   RMSNorm    │
│  Output Head │  Weight-tied to embedding (float)
└──────┬──────┘
       │
       ▼
   Logits (10,000 vocab)

Component Details

BitLinear (Ternary Weights)

All linear projections in the model body use ternary quantization with a straight-through estimator:

alpha = mean(|W|)
W_ternary = clamp(round(W / alpha), -1, +1) × alpha

During training, gradients flow through the quantization via the straight-through estimator (STE). During inference, weights can be stored as 2-bit integers and all multiplications become additions, subtractions, or zeros.

GatedConvMixer (Token Mixing)

Instead of self-attention, FlashLM v4 uses a gated causal depthwise convolution:

gv = BitLinear_up(x)           # project to 2*dim
gate, val = split(gv)          # split into gate and value
gate = sigmoid(gate)
h = CausalDepthwiseConv1D(val, kernel_size=8)
output = BitLinear_down(h * gate)

This gives the model a receptive field of 8 tokens per layer, or 48 tokens across all 6 layers. The operation is O(T) in sequence length with no quadratic attention cost.

TernaryGLU (Channel Mixing)

The feed-forward network uses a gated linear unit with ternary weights:

output = BitLinear_down(SiLU(BitLinear_gate(x)) * BitLinear_up(x))

Dimensions: 192 → 512 → 192 (expansion ratio ≈ 2.67x).

RMSNorm

Root Mean Square Layer Normalization (no bias, no mean subtraction) before every sub-layer.

Weight-Tied Embedding

The input embedding and output projection share the same weight matrix (float32, 10,000 × 192). This is the only float component in the model; everything else is ternary.

---

Training Configuration

---

Training Loss Curve

The loss curve shows no sign of plateauing. Extended training on more powerful CPU hardware is expected to push BPC significantly lower.

---

Inference

At inference time, the ternary weights mean the core model operations reduce to:

• Weight = +1: add the activation
• Weight = -1: subtract the activation
• Weight = 0: skip (no operation)

No floating-point multiplications are needed in the model body. The only float operations are the embedding lookup, RMSNorm scaling, and the tied output projection.

---

Sample Generations

From the final checkpoint (step 5199):

Prompt: "Once upon a time"

Once upon a time, there was a little girl named []. She loved to play outside and explore the world. One day, she wanted to go outside. She went to the [] and saw a big tree. She wanted to catch it, but the [] was too small.

Prompt: "[] and his mom went to the"

[] and his mom went to the []. They had lots of fun and [] each other. And they never gave up. Once upon a time, there was a little girl called []. She loved to explore and find new things.

([] indicates UNK tokens from the 10K vocab limitation — fixable by expanding vocabulary)

---

Limitations

• Vocabulary coverage: The 10K token vocabulary doesn't cover all words in TinyStories. Some outputs contain UNK tokens. Expanding to 15-20K tokens would fix this.
• Training data: Only 10.6M of TinyStories' 470M tokens were seen. The model is significantly undertrained.
• Receptive field: The 48-token effective receptive field (8 kernel × 6 layers) limits long-range coherence compared to attention-based models.
• Domain: Trained exclusively on children's stories. Not a general-purpose language model.

---

What's Next

• Extended training on a Ryzen 7950X3D (16 cores, 96MB V-Cache, 128GB RAM) for multiple days
• Scaled variant (~15M params, d=384, 8 blocks) with fixed tokenizer (frequency-based vocab)
• Standalone training script release (MIT license)
• Target: close the BPC gap with TinyStories-1M through longer training and proper vocab coverage

---

Previous Versions

• FlashLM v3 (13.6M) — ternary weights on FineWeb-Edu, val loss 6.80, incoherent output
• FlashLM v3 Demo — interactive demo of v3

---

Citation

@misc{flashlm-v4-bolt,
  title={FlashLM v4 "Bolt": A Ternary Language Model Trained on CPU},
  author={Cheng Chang},
  year={2026},
  url={https://huggingface.co/changcheng967/flashlm-v4-bolt}
}

---

License

MIT