NanoFable-28M-ternary

NanoFable-28M-ternary

A 27.8M-parameter decoder-only transformer trained on TinyStories with ternary (1.58-bit) in-block weights — one point on the coherence-vs-bytes frontier study (spec).

The Hub's Safetensors panel reports F16. That is the storage dtype, not the weight precision. The in-block linear weights in model.safetensors take exactly three distinct values — scale × {-1, 0, +1} — but are stored dequantized into fp16 so any standard fp16 loader reads them with no custom code. The actual 1.58-bit packed form is model.tpack, and that is what the byte figures below measure.

Training

TinyStories, single GPU (T4), fp16 autocast. AdamW (β 0.9/0.95, wd 0.1), peak LR 3e-4, cosine to 10% after 3% warmup, 65,536 tokens/step (~7,630 steps). All hyperparameters frozen across the 16-run sweep (frozen config).

Evaluation

Val perplexity on held-out TinyStories + LLM-judge coherence (Qwen2.5-7B-Instruct, frozen rubric, 200 frozen prefixes, greedy decoding; instrument calibration in calibration.md — reference anchor: TinyStories-8M at 4.232).

• Val PPL: 13.37 / 13.32 (seed 0 / seed 1)
• Judge score (0–5, greedy, n=400 pooled): 1.87 ±0.07 (grammar 2.98 · consistency 1.80 · completes 0.82)
• Capability gates: coherence (≥4.0) fail · PPL (≤8.09) fail · T1 surface-fluency (grammar ≥4.55, TS-8M anchor) fail — no sweep config cleared the coherence or T1 gates (pre-registered null; see eval/calibration.md)

Usage

Weights ship as model.safetensors (fp16). Ternary in-block linears are stored dequantized — scale * {-1,0,+1} already materialized in fp16 — so both arms load into the same fp16 model class.

from safetensors.torch import load_file
from nanofable.config import TIERS
from nanofable.model import build_model
from nanofable.tokenizer import load_tokenizer
from nanofable.generate import generate

model = build_model(TIERS["large"], "fp16")
model.load_state_dict(load_file("model.safetensors"), strict=False)  # lm_head re-ties to tok_emb
model.eval()

tok = load_tokenizer("tokenizer.json")
print(generate(model, tok, "Once upon a time", temperature=1e-4, top_k=0))

Repository layout

model.safetensors         # seed 1 — the weights this card describes
config.json
model.tpack               # packed browser payload (the compressed artifact)
meta.json  eval.json  metrics.csv
tokenizer.json            # 4,096-vocab custom BPE
seed0/                    # second training replica, same files

Both training replicas are published. The root weights are seed 1, selected by judge score. Do not read that as a quality ranking — the two seeds are statistically indistinguishable: their 95% judge CIs overlap, and val PPL disagrees with judge score on which seed leads in most configs of this sweep. The choice of which sits at the root is effectively arbitrary.

model.tpack is the packed browser payload (pack format v1), with in-block linears at 1.58 bits/weight. This is the artifact the byte accounting above refers to, and what the demo site streams.

Note that model.safetensors is not the compressed form — it stores the ternary weights dequantized (scale × {-1,0,+1} materialized back into fp16), so it is the same size as its fp16 twin. That is deliberate: it keeps the weights loadable by any standard fp16 model class. The compression is real, but it lives in the .tpack.

Training checkpoints (optimizer state, fp32 latent weights) are not published here; they live in the source repo.

Limitations

Children's-story English only (TinyStories distribution) — not a general language model. 512-token context. Research artifact of a controlled precision study; not safety-tuned.

The byte figure is the spec §6 packed accounting; checkpoints on disk store fp32 latent weights and are larger.