Downloads · 30 days
0
baseline-nanochat-d24
kz919/baseline-nanochat-d24
baseline-nanochat-d24 is a text generation model from kz919. Use it when you need the model to write or continue text. It is set up for nanochat. The card lists the license as mit.
The stock nanochat architecture (@ 92d63d4) trained at the d24 (~1.38 B param) scale. This is the baseline in a controlled comparison against a two‑pass routing‑loop variant (kz919/looped-nanochat-d24).
Access
Public
Updated Jul 18, 2026
Repo size
4.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt4.2 GB · 100%
At a glance
- Task
- Text Generation
- Library
- nanochat
- License
- mit
- Access
- Public
- Created
- Jul 18, 2026
- Updated
- Jul 18, 2026
- SHA
- dd54d655
Try a prompt
Model card
From the Hugging Face model README
baseline-nanochat-d24
The stock nanochat architecture
(@ 92d63d4) trained at the d24 (~1.38 B param) scale. This is the baseline
in a controlled comparison against a two‑pass routing‑loop variant
(kz919/looped-nanochat-d24).
Code: https://github.com/kyleliang919/looped_nanochat
Results (vs. looped, identical training budget)
| Metric | Baseline | Looped |
|---|---|---|
| Base CORE | 0.2552 | 0.2783 |
| ChatCORE (full sets) | 0.2253 | 0.2521 |
| ARC‑Easy | 62.04% | 68.01% |
| ARC‑Challenge | 50.00% | 52.22% |
| MMLU | 37.06% | 37.96% |
| GSM8K | 2.88% | 4.17% |
| HumanEval | 10.98% | 10.98% |
The looped model wins or ties every metric at ~2.4× the training compute. See the full report.
Model details
| Layers | 24 |
n_embd | 1536 |
| Heads / KV heads | 12 / 12 |
| Vocab | 32768 |
| Context | 2048 |
| Window pattern | SSSL |
| Routing loop | off (standard nanochat) |
| Params | ~1.38 B |
| Precision | bf16 (fp8 matmuls in training) |
Training: pretrained for 5,568 steps on ~5.84 B tokens
(--target-param-data-ratio=8, 1,048,576‑token batches), then chat‑SFT on
SmolTalk + MMLU(×3) + GSM8K(×4). This checkpoint is the SFT model (step 467).
Files
model_000467.pt— model weights (state_dict)meta_000467.json—model_config+ training configtokenizer.pkl,token_bytes.pt— nanochat tokenizer
Usage
Uses the nanochat runtime (not 🤗 Transformers). Place the files into
~/.cache/nanochat/chatsft_checkpoints/baseline-d24/ and
~/.cache/nanochat/tokenizer/, then:
python -m scripts.chat_cli -g baseline-d24 -i sft -p "Why is the sky blue?"
Limitations
A ~$100‑tier micro‑model: it hallucinates and is weak at math/code. Research artifact, not for production.
License
MIT (inherited from nanochat).
- Task
- Text Generation
- Library
- nanochat
- License
- mit
- Languages
- en
- Created
- Jul 18, 2026
- Updated
- Jul 18, 2026