NanoFable-28M-fp16

NanoFable-28M-fp16

A 27.8M-parameter decoder-only transformer trained on TinyStories with fp16 weights — one point on the coherence-vs-bytes frontier study (spec).

Training

TinyStories, single GPU (T4), fp16 autocast. AdamW (β 0.9/0.95, wd 0.1), peak LR 3e-4, cosine to 10% after 3% warmup, 65,536 tokens/step (~7,630 steps). All hyperparameters frozen across the 16-run sweep (frozen config).

Evaluation

Val perplexity on held-out TinyStories + LLM-judge coherence (Qwen2.5-7B-Instruct, frozen rubric, 200 frozen prefixes, greedy decoding; instrument calibration in calibration.md — reference anchor: TinyStories-8M at 4.232).

• Val PPL: 5.94 / 5.39 (seed 0 / seed 1)
• Judge score (0–5, greedy, n=400 pooled): 3.11 ±0.09 (grammar 3.83 · consistency 3.23 · completes 2.26)
• Capability gates: coherence (≥4.0) fail · PPL (≤8.09) pass · T1 surface-fluency (grammar ≥4.55, TS-8M anchor) fail — clears the PPL gate; no sweep config cleared the coherence or T1 gates (pre-registered null; see eval/calibration.md)

Usage

Weights ship as model.safetensors (fp16). Ternary in-block linears are stored dequantized — scale * {-1,0,+1} already materialized in fp16 — so both arms load into the same fp16 model class.

from safetensors.torch import load_file
from nanofable.config import TIERS
from nanofable.model import build_model
from nanofable.tokenizer import load_tokenizer
from nanofable.generate import generate

model = build_model(TIERS["large"], "fp16")
model.load_state_dict(load_file("model.safetensors"), strict=False)  # lm_head re-ties to tok_emb
model.eval()

tok = load_tokenizer("tokenizer.json")
print(generate(model, tok, "Once upon a time", temperature=1e-4, top_k=0))

Repository layout

model.safetensors         # seed 1 — the weights this card describes
config.json
model.tpack               # packed browser payload (what the demo site streams)
meta.json  eval.json  metrics.csv
tokenizer.json            # 4,096-vocab custom BPE
seed0/                    # second training replica, same files

Both training replicas are published. The root weights are seed 1, selected by judge score. Do not read that as a quality ranking — the two seeds are statistically indistinguishable: their 95% judge CIs overlap, and val PPL disagrees with judge score on which seed leads in most configs of this sweep. The choice of which sits at the root is effectively arbitrary.

model.tpack is the packed browser payload (pack format v1), fp16 throughout. This is the artifact the byte accounting above refers to, and the one the demo site streams for in-browser inference. It carries the same weights as model.safetensors, in the container the browser runtime reads.

Training checkpoints (optimizer state, fp32 latent weights) are not published here; they live in the source repo.

Limitations

Children's-story English only (TinyStories distribution) — not a general language model. 512-token context. Research artifact of a controlled precision study; not safety-tuned.