disk-routed-chat-0.5b-v2

Disk-Routed Chat 0.5B — v2 (frontier training stack)

Experimental research artifact. A 0.5B disk-routed recurrent language model designed to run inference on CPU + SSD (2 rows/token, constant per-token I/O), trained with a frontier efficiency stack. This card documents an efficiency result, not a production assistant.

What is interesting here (the training result)

Trained from scratch on a single RTX 4090 in ~12 hours for ~$4:

Frontier training stack

• Chunk-parallel linear recurrence — turns the token-by-token scan into batched matmuls.
• torch.compile — fuses the scan; 3.1× speedup (32k → 100k tok/s bench), the decisive lever.
• Multi-token prediction (extra head) — sample efficiency.
• Textbook/synthetic data (phi-1 principle) — quality per token.
• WSD schedule (MiniCPM) + Muon optimizer (~2× vs AdamW).
• Disk-routed product-key memory — 2 rows/token, capacity on SSD.

Honest limitations

The base model produces coherent English, but the chat model is minimal and weak: it learned the assistant format but does not reliably answer greetings or simple questions (e.g. "capital of France" is wrong). This is the expected ceiling of 0.5B / 3B-tokens / ~77M-active. Use it as a systems/efficiency reference, not a usable chatbot.

Files

• base.pt — pretrained base (config + weights)
• sft.pt — after supervised fine-tuning (chat format)
• model_frontier.py — model definition (DiskChatMTP, chunk-parallel scan)
• fineweb-tokenizer.json — 32k BPE tokenizer

Load

import torch
from model_frontier import DiskChatMTP
ck=torch.load('sft.pt',map_location='cpu')
m=DiskChatMTP(**ck['config']); m.load_state_dict({k.replace('_orig_mod.',''):v for k,v in ck['model'].items()}); m.eval()

Roadmap: distillation (MiniLLM), Rho-1 selective tokens, FP8, and a CPU/SSD int8 runtime for this architecture.

CPU + Disk inference throughput (measured)

Chat decode running on CPU only (no GPU), with the 422M-param product-key memory table stored as int8 on disk (memory-mapped) and only 2 rows read per token. 10 runs on an AMD EPYC 7713, 16 threads:

The recurrent backbone keeps a constant-size state (no KV cache), so decode cost is constant per token regardless of context length, and capacity lives on disk.

SSD-only inference (C + O_DIRECT — true cold disk)

The previous CPU section used PyTorch with the table mmap-ed, so the 422 MB table got cached in hot RAM (compute-bound, 43.9 tok/s). For a faithful SSD-only measurement we use a hand-written C runtime that reads the int8 product-key table with O_DIRECT (bypasses the OS page cache), so the 2 rows/token are genuinely fetched from the SSD each step. Controller weights stay in RAM; the table never loads into RAM.

~4× faster than the framework path, at <300 MB RAM with the model's capacity resident on SSD — reading a constant 8 KiB/token regardless of table size. Runtime: direct_chat2.c (recurrent constant-state decode, O_DIRECT paging, OpenMP).

Post-SFT update (extended 2-hour SFT)

The chat model was fine-tuned further: 2 hours on 100k UltraChat dialogues (vs the initial 25k). File: sft2.pt. Quality improved noticeably on simple questions while throughput is unchanged (same architecture/size).

Quality (post-SFT samples)

Still an experimental minimal-chat model (open greetings remain weak), but factual recall on simple questions is better than the 25k-SFT version.

Speed (post-SFT, 10 runs each, AMD EPYC 7713, no GPU)

Confirms the disk-routed inference profile is stable after alignment: ~184 tok/s at <300 MB RAM, capacity on SSD, no GPU.

Re-trained SFT2 (2026-07-20)

sft2.pt was re-trained from base.pt on a rented RTX 4090: 100k UltraChat dialogues, ~2h, 12898 steps, loss 2.58 → 2.16 (ppl 13.1 → 8.6), assistant-only loss mask. Ran eager (no torch.compile — its graph can't trace the chunk-parallel scan's data-dependent loop, and the fallback desyncs gradient checkpointing), so the 3.1× compile lever is not active in this run.

Verified coherent, prompt-appropriate generation (factually loose, as expected at 0.5B / 77M-active):

USB / SSD runtime files are under runtime/: controller.bin (~294 MB, RAM-resident), mem.i8 (~422 MB int8 product-key table, read 2 rows/token via O_DIRECT), scale.txt. Swap guide for an existing USB deploy: USB_UPDATE_GUIDE_sft2.md.

SFT3 — coherence fix (2026-07-20)

sft2.pt still rambled off-topic on short prompts (it was trained on long, verbose UltraChat essays). sft3.pt continues training from sft2.pt on Alpaca-cleaned (51,760 short, direct instruction→answer pairs) — teaching concise, on-topic answers. 90 min on a rented RTX 4090, EMA-best checkpoint, best EMA loss 1.97 (ppl 7.2 vs sft2's 8.6). Same architecture/size/throughput.

Coherence before → after (same prompts, greedy/light-sampling):

Still 0.5B / 77M-active — coherent and prompt-appropriate, but factually loose (e.g. "2+2 = 3"). The runtime/ files and USB_UPDATE_GUIDE_sft2.md procedure now target sft3.pt; sft2.pt is kept for history.

SFT4 — data diversity (2026-07-20) — recommended checkpoint

sft4.pt continues from sft3.pt on Alpaca-cleaned + Dolly-15k (66,771 records; Dolly is human-written, adding QA / brainstorm / summarize / closed-QA variety). 90 min on a rented RTX 4090, EMA-best. Best EMA loss 2.11 (ppl 8.3) — higher than sft3's 7.2 because the mixed distribution is broader, not worse (loss across more task types).

Improves greetings and factual QA over sft3:

Still 0.5B / 77M-active: coherent and prompt-appropriate, factually loose (math still wrong). The runtime/ files and USB_UPDATE_GUIDE_sft2.md procedure now target sft4.pt; sft2.pt/sft3.pt are kept for history.

Storage provenance for the 184.4 tok/s reference

The 184.4 tok/s result above was measured with 16 threads on an AMD EPYC 7713 and a fast server-local SSD under Linux O_DIRECT. The exact server SSD model, interface, IOPS, and latency were not recorded. Therefore it must not be presented as a Micron 3400 result or as a guaranteed result on any other host.

For a useful target-host comparison, the laptop used for the removable-media test has two Micron 3400 MTFDKBA512TFH 512 GB NVMe drives, configured as Linux mdraid RAID 0 with an ext4 root volume (~953.6 GiB usable). The following are manufacturer per-drive specifications supplied for the 512 GB model, not a MONKE benchmark:

MONKE at the 184.4 tok/s reference requires only about 369 4 KiB random reads/s (2 rows/token). These Micron specifications show ample nominal storage headroom, but sequential MB/s and vendor random-IOPS figures often use different queue depths than MONKE's latency-sensitive read pattern. The laptop's CPU, RAID behavior, filesystem/direct-I/O behavior, thermals, and actual QD1 random-read latency still determine achieved tok/s. A valid Micron result requires running the same 10-run C + O_DIRECT benchmark with mem.i8 on that array.

Removable-media result: generic USB 2.0 flash drive

A MONKE flash-drive deployment was measured on a generic/unbranded 128 GB USB 2.0 flash drive, mounted on Linux as exFAT (/run/media/jaime/ECF5-C242). Usable capacity was 117.2 GiB. The app measured 45–48 tok/s (about 46.5 tok/s midpoint) on this deployment.

This is not a bandwidth-bound workload: at 48 tok/s, model traffic is only ~384 KiB/s. The critical metric is low-queue-depth 4 KiB random-read latency / IOPS, where commodity USB 2.0 flash is weak. Typical USB 2.0 sequential read figures (~15–35 MB/s) therefore do not predict MONKE decode speed. Host CPU also matters; the 184.4 tok/s reference used 16 threads on an AMD EPYC 7713.

Storage ladder toward the 184.4 tok/s reference

Do not read this ladder as guaranteed tok/s predictions. The Kingston and SSK numbers are manufacturer sequential specifications, not MONKE results. A drive reaches the reference only if its random 4 KiB latency/IOPS and host CPU match the reference workload. Test the same model on the target host's internal NVMe first; the delta against the removable drive isolates storage from CPU.

Safe Linux storage check (read-only; uses the table file):

fio --name=monke-random-read \
  --filename=/run/media/jaime/ECF5-C242/MONKE/model/mem.i8 \
  --rw=randread --bs=4k --iodepth=1 --numjobs=1 \
  --time_based --runtime=30 --readonly --invalidate=1

MONKE needs about 2 × tok/s 4 KiB random-read IOPS: ~96 IOPS at 48 tok/s and ~369 IOPS at the 184.4 tok/s reference.