ming-vintage-scratch-30m

ming-vintage-scratch-30m

A from-scratch char-level nanoGPT trained on a pre-1424 Classical Chinese corpus. The first non-English entry in the vintage LLM category.

2026-08-24 audit. Two claims previously made here were wrong. Leakage is not zero by construction — measured at 0.6% language leakage, 0.0% out-of-corpus characters, and the vocabulary is not purely Han. And the corpus is not purely pre-1424: 18.58% of documents carry an 18th-century editorial layer that this model reproduces at up to 10%. See Corpus contamination.

29.74M params · 31,344 char vocabulary · 477M training tokens · 0.68 epochs · trained on Mac M4

Quick Start

git clone https://huggingface.co/Beltran12138/ming-vintage-scratch-30m
cd ming-vintage-scratch-30m
python sample.py --start="如是我聞" --max_new_tokens=200 --temperature=0.8 --top_k=200

Requires torch, numpy. No CUDA needed — runs on CPU or MPS.

What Is This?

This is a from-scratch vintage language model trained on pre-1424 Classical Chinese — the treatment arm of a controlled experiment.

The experiment: Can a LoRA adapter make a modern LLM "vintage," or do you need to train from scratch to truly lock the world-model?

The control arm: ming-vintage-qwen3b-lora — a LoRA fine-tuned on the same pre-1424 corpus, applied to Qwen2.5-3B-Instruct.

Finding: LoRA transfers register but the base model's multilingual tokens leak through as a backdoor (Turkish İ, Arabic لأسم, Korean 염, programming terms like CriticalSection all appear in LoRA outputs). The from-scratch model shows near-zero language leakage — 0.6% across 160 generations, against 35–69% for the base.

Correction (2026-08-24). This section previously claimed the scratch model had zero leakage by construction and had never seen any language but pre-1424 Classical Chinese. Both are false. Its 31,344-character vocabulary contains 47 Latin letters, 3 Hangul syllables, 55 katakana, 10 hiragana, Greek, Cyrillic and a run of Myanmar — the corpus contains Japanese commentary, Russian Dunhuang shelfmarks and IAST Sanskrit transliteration. The low leakage is probabilistic, not constructive. See Corpus contamination.

This is the 4th instance of the "不在 X 在 Y" (Not in X, but in Y) thesis: style lives in the adapter weights; the world-model lives in the full parameter space.

Training Data

• Source: kanripo corpus (Buddhist canon + Confucian classics + historical texts), pre-1424 CE cutoff
• Corpus size: 162,849 documents, 478M characters
• Vocabulary: 31,344 unique characters (char-level tokenization)
• Final tokens: train 477M / val 0.95M (uint16 numpy arrays)

Character-level tokenization was chosen because each Classical Chinese character carries independent semantic weight — subword tokenization (BPE) would fragment individual characters and destroy the register signal.

Training Procedure

Key hardware lesson: On M4 MPS with 16GB, block_size=384 stalls at iter 1 (activation memory ∝ block²). Reducing to block_size=256 while keeping 30M parameters is the correct tradeoff — parameters determine model quality, block size only affects context window.

The 40K checkpoint is recommended over the 80K — the additional 40K steps with constant LR caused the model to overfit toward the majority class (Buddhist texts, ~70% of corpus) at the expense of minority registers (Confucian, historical).

Evaluation: 6-Seed Continuation Probe

Since this is a raw continuation model (not instruction-tuned), evaluation uses continuation seeds rather than QA prompts. Six seeds were tested on both the scratch model and the LoRA control (temperature=0.8, top_k=200, max_tokens=200).

Three-Way Comparison

Key Finding: The Multilingual Backdoor

The LoRA adapter was trained to suppress the base model's modern knowledge and boost Classical Chinese register. It succeeds partially on register — but Qwen2.5-3B was trained on 100+ languages, and the LoRA adapter (r=16, ~51MB) covers only a fraction of the vocabulary.

Rare-language tokens bypass the vintage filter. Turkish İ, Arabic لأسم, Korean 염, and English programming terms (CriticalSection) all appear in LoRA-generated "Classical Chinese" text. These tokens are low-frequency enough that the LoRA adapter never learned to suppress them — but they remain in the vocabulary and can be sampled.

This is a structural limitation of adapter-based vintage models: any base model trained on multiple languages has latent pathways that a domain-specific adapter cannot fully close.

The from-scratch model reduces this to 0.6% (one Latin letter across 160 generations). It does not eliminate it by construction: its 31,344-character vocabulary is not purely Han, and includes Latin, Hangul, kana, Greek, Cyrillic and Myanmar characters inherited from the corpus. The suppression is statistical.

Relation to talkie-1930

This project draws methodology from talkie-1930 (Levine, Duvenaud, Radford), which demonstrated that from-scratch pre-1931 English training produces a coherent vintage world-model. Key findings from talkie that informed this work:

• "Controlling for question anachronism, the performance gap roughly halves" — the same principle applies: QA probe format with mixed punctuation was itself anachronistic for the scratch model.
• From-scratch is the only way to lock the world-model — talkie's 260B token corpus achieved this for English; ming-scratch attempts it for Classical Chinese at a ~1000× smaller scale.

Where talkie targets production-scale vintage English (13B params, 260B tokens), ming-scratch is a nano proof-of-concept for vintage Classical Chinese (30M params, 477M tokens).

Ablation: Scaling Study (109M / block 512)

To test whether the repetition ceiling is a capacity/context limitation, we trained a larger variant on the same corpus:

Finding: Despite a 3.6x parameter increase and 2x context window, val loss dropped by 0.71 but coherence length did not meaningfully improve. The 200K-step run overfit (val 3.61 -> 3.75 in the final 14K steps), indicating the 460M-char corpus was exhausted at ~1.7 epochs — well below the Chinchilla-optimal ~2.2B tokens for 109M params.

Implication: The repetition ceiling is structural to char-level modeling at this data scale, not a capacity or context-window bottleneck. Repetition occurs at 20-60 chars — far within the 256-char block — confirming context length was never the constraint. Scaling parameters further without proportionally more data, or adopting a subword tokenizer, is required for further gains. The 30M v0 is therefore the honest representative of what weekend-scale char-level vintage Chinese can achieve.

The v2 checkpoint is archived locally for reproducibility but not published.

Corpus contamination (2026-08-24 audit)

A full audit of the training corpus (162,849 documents / 477,824,316 characters) found that 18.58% of documents carry a Qing-dynasty editorial layer — the 1781 Siku Quanshu commissioners' preface, one template repeated ~1,750 times:

The corpus filter tested "was the author alive before 1424" (a metadata field), not "does this text contain only pre-1424 content". A Song gazetteer transmitted in the Qianlong recension passes, and brings its 1781 preface along.

This model reproduces that layer. Three characters of prompt (恭校上) retrieve four Qing commissioners by name:

seed:   恭校上
output: 總纂官(臣/)紀昀(臣/)陸錫熊(臣/)孫士毅總校官(臣/)陸費墀欽定四庫全書…

Rate by seed type: template fragments 10.0%, ordinary book openings (史部/巻一/原序) 0.8%, in-period text openings 0/200. The base model and the LoRA in raw completion both score 0/144 — they have no idea who these people were.

From-scratch training immunises a model against someone else's contamination and amplifies your own. A 30M model learns a template repeated 1,750 times before it learns anything else.

Full data, scripts and per-generation records: FINDINGS.md

Limitations

1. Size: 29.74M parameters. Output degrades into repetition after 20-50 characters. This is a nano proof-of-concept.
2. Corpus imbalance: ~70% Buddhist texts. Confucian and historical registers are underrepresented.
3. Block size: 256 characters (constrained by M4 MPS 16GB). Cannot model long-range dependencies.
4. Data volume: 477M tokens ≈ 0.34 epochs — below Chinchilla-optimal (~20 tokens/param). Undertrained.
5. Char-level only: Out-of-vocabulary characters mapped to token 0.
6. Mac-only tested: Training pipeline assumes macOS MPS.
7. Corpus is not purely pre-1424: 18.58% of documents carry an 18th-century editorial layer, which the model reproduces at up to 10% depending on the prompt. See Corpus contamination.
8. Vocabulary is not purely Han: 47 Latin letters, 3 Hangul, 55 katakana, Greek, Cyrillic and Myanmar characters are present. Language suppression is statistical (0.6%), not constructive.

Citation

@misc{ming-vintage-scratch-30m,
  author = {Beltran12138},
  title = {ming-vintage-scratch-30m: A From-Scratch Vintage Classical Chinese Language Model},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Beltran12138/ming-vintage-scratch-30m}}
}

See Also

• Control arm: ming-vintage-qwen3b-lora
• Inspiration: talkie-1930-13b-it
• List: awesome-vintage-llms