cpuflow-v8-discrete

CPUFlow v8 — Discrete State Streams

Best PPL in the CPUFlow series (9.30) but generates incoherent text. Uses hard argmax routing with straight-through estimator.

Results

Architecture

embed + CumStepPos → [RouteBlock × 6] → LayerNorm → tied output + FSP

RouteBlock:
  x_n = LayerNorm(x)
  slot_id = argmax(W_route(x_n))    # hard routing via STE
  slot_state = read_write(slot_id, x_n)
  x = x + W_out(slot_state)
  x = x + ff_down(relu(ff_up(LayerNorm(x))))

Each token routes to exactly one of 32 memory slots via argmax. Straight-through estimator allows gradient flow through the discrete choice.

Why incoherent?

v8 has the best PPL (9.30) but produces word salad. The hard slot routing disrupts the continuous context representation that the cumsum backbone provides. PPL measures token prediction accuracy, not logical coherence. v5-LN (PPL 11.94) generates more coherent text because its cumsum maintains a running summary of all past tokens.

Limitations

• Completely incoherent generation despite best PPL
• PPL 9.30 is misleading — this model cannot tell a coherent story
• Hard routing prevents smooth context flow between positions
• This is a research model demonstrating the PPL ≠ coherence finding

Usage

import torch
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
checkpoint = torch.load("best.pt", map_location="cpu")
# Build model (see train_cpuflow_v8_discrete.py for full architecture)

See GitHub for full training code.

Citation

@misc{Chang,
  title        = {FlashLM: CPU-Native Language Models Trained From Scratch on Free-Tier Hardware},
  author       = {Chang, Cheng},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.20113960}
}

MIT License.