leda-tinystories-15m

Leda TinyStories 15.7M

A small English story-continuation model trained from scratch by Brandon Ling using Spar, a custom C++23 deep learning framework. Leda source.

This is the final GPT-4-only September 8, 2026 release, not the earlier 40M model. It continues story openings; it is not an instruction-following or chat assistant.

Run locally on CPU

Download this repository to a directory, then from that directory run:

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python inference.py --prompt "One day, a little fox found a key in the forest." --max-new-tokens 96 --temperature 0

For sampling, use --temperature 0.8 --seed 42. Temperature 0 is greedy. Sampling uses the full temperature-scaled distribution with no top-k or top-p filter and a seeded SplitMix64 inverse-CDF sampler. The script prints text, exact token IDs, counts, and finish: "eod" or "length".

The supplied inference.py is an explicit, standalone PyTorch CPU implementation using safetensors. Transformers AutoModel, a hosted inference API, and automatic Hugging Face inference-widget compatibility are not implemented or claimed. Read the script before running it. This reference recomputes the complete prefix each step; the browser runtime uses a KV cache.

The 256-token context includes the opening and continuation, including an end token if produced. Requests that do not fit are rejected, never silently truncated. ByteLevel BPE uses 4,096 text tokens; ID 4096 is a model-only end-of-document token and must not be decoded as text. No normalization, prefix space, padding, or implicit special tokens are added during inference.

Model and provenance

• 15,734,784 parameters; 6 layers; width 384; MLP width 1,536; 6 query and 6 key/value heads; head dimension 64.
• SwiGLU, RMSNorm (epsilon 1e-5), per-head QK RMSNorm (epsilon 1e-6), RoPE (theta 10,000), and tied input/output embeddings.
• FP32 weights. Projection matrices retain the native input-by-output layout.
• Trained on 2,707,825 prepared GPT-4-generated TinyStories stories from dataset revision f54c09fd23315a6f9c86f9dc80f725de7d8f9c64.
• 554,547,990 predicted targets over 67,960 updates; 37 uncovered tail tokens. Context 256, microbatch 4 with accumulation 8; final partial update 2,550 targets.
• Training text preparation normalized CRLF/CR to LF, trimmed outer whitespace, and collapsed repeated paragraph newlines. Training-data normalization is not silently applied to inference input.
• Source corpus: TinyStories. Training completed; this release required no new training.

Checkpoint SHA256: f3012c8a389f98ef61771a06dd281ddeeaac814a1996f53b7a61aebdcb3cc033. Tokenizer SHA256: 57881206f1bbe864868f92e77b632f65a5fbba0195d92c4828805025318ee851. model.safetensors contains the 68 finite model tensors extracted from that checkpoint. model.sparckpt preserves the exact native checkpoint; model-config.json describes its non-Transformers layout. SHA256SUMS.json records package file hashes.

Evaluation and limitations

The final review read 24 development and 24 sealed confirmation continuations. The model produces readable short stories but sometimes repeats phrases, loses the original problem, confuses character roles or ownership, and makes unsupported causal jumps. Sampled outputs are uneven. This was a small qualitative review, not a blinded benchmark or statistical quality claim.

The model emitted EOD in 13/24 development and 12/24 confirmation outputs; the others reached the 128-new-token cap. Final monitoring loss was 1.26444. The separate 26,604 reserved validation stories were not scored; no held-out NLL result is claimed. Old-model validation data differed, so endpoint losses are not comparable quality evidence. More context space is not evidence of better consistency.

Treat generated stories as model output, not reliable advice or factual statements. There is no chat alignment or safety-tuned assistant behavior.

Browser playground

The accompanying playground uses the final Float32 weights with WebGPU and a WebAssembly CPU fallback, not the old leda/web artifact. It streams tokens, retains actual logits at each generated position, and branches using exact token IDs. Raw probabilities mean softmax(logits); the decoding distribution is softmax(logits/temperature), or a point mass on the greedy argmax. Manually forced branch tokens are marked as such. No sampling filters are applied.

Runtime checks compare native logits, probabilities, greedy continuations including EOD, and branched prefixes. Device-dependent floating-point differences may change a choice when alternatives are close; seeds are not a promise of cross-device bitwise identity. Validation measurements are provided with the playground source.