Summer-0.5B-Chat

Summer-0.5B-Chat

Summer-0.5B-Chat is a from-scratch bilingual (Chinese/English) chat model: 524,336,128 parameters, trained from random initialization on 14.6B + 625M chat tokens with a self-trained 81,903-piece tokenizer.

It is not a fine-tune or a tokenizer-swap of any existing model. The architecture follows Qwen/Qwen3-0.6B-Base (28 layers / hidden 1024 / GQA 16:8 / head_dim 128 / tied embeddings / RoPE theta 1e6), but every weight starts from N(0, 0.02).

Stage

Chat (v9e2) — single-stage post-training from Summer-0.5B-S0 (not from Summer-0.5B-S1; the parallel-data anneal is a separate branch for the translation downstream, unrelated to becoming a general chat model).

This follows nanochat's current recipe (chat_sft.py, commit 1ddaad1 onward): no separate mid-training stage. An earlier version of this project's chat line used a two-stage midtrain-then-SFT design copied from an older nanochat snapshot; that stage was dropped once we confirmed nanochat itself had merged it into a single SFT pass, relying on a large enough data mixture to teach the special tokens (<user>/<assistant>/<end>) instead of a dedicated full-loss warm-up stage.

• Data: chat_v9 mixture, 305,187 rows / 625M tokens — the same chat_full mixture as the previous (v8) release (SmolTalk full 460K, MMLU-aux x1, ARC-Easy/Challenge, GSM8K x4, spelling tasks, Chinese data scaled to keep row-share constant), plus three targeted additions made after manual testing of v8 surfaced specific, reproducible failures:
  • Identity 2x1,000 → 2x5,000 (same hand-written QA pool, just seen more times during training) — v8 showed inconsistent self-identification across turns; the identity signal was only 0.23% of the mix by row count.
  • WildChat-1M Chinese subset, 8,000 rows (real user↔ChatGPT conversations, filtered for toxicity and for any turn where the assistant self-identifies as ChatGPT/GPT/OpenAI, which would otherwise teach the wrong identity) — v8's data was 100% synthetic QA/MC-style tasks with no coverage of casual, non-question follow-ups ("continue", a passing remark), which v8 handled by drifting to an unrelated topic.
  • ALMA parallel corpus reformatted as bidirectional translation instructions, 6,000 rows — v8 was trained entirely on Summer-0.5B-S0 (monolingual), so the chat model had never seen a translation instruction; asking it to translate produced fabricated, off-topic text rather than an attempt.
• Packing: BOS-aligned best-fit, padding (not cropping) at row boundaries so no conversation is ever cut mid-turn. Loss is computed on assistant turns only.
• Learning rate: single linear decay from 0.8x the pretraining peak LR to 0, matching nanochat's init_lr_frac=0.8, spread across the full training run below (not decayed-then-restarted).
• 9,536 steps = 2 epochs over the same chat_v9 data (v9 itself was 1 epoch, 4,768 steps). At 524M params / 625M post-training tokens (~1.2 tokens/param), there was headroom to see the data twice without the val loss plateauing — it kept dropping past where the 1-epoch run ended (1.2459 → 1.1932 final).

Every input must start with <bos>, same as Summer-0.5B-S0.

What to expect

This is a 14.6B + 625M chat-token model. For scale: Qwen3-0.6B-Base saw 36T tokens — about 2,700x more. Treat the numbers below as what that budget buys, not as a competitive result.

Letter multiple-choice (nanochat's primary format)

A second epoch over the identical data raised every letter-MC number by 0.016-0.044 — a bigger jump than the v8→v9 data changes produced, and well outside sampling noise (e.g. MMLU's 14,042-item standard error is ~0.004; the observed gap is 4x that). Makes sense: repeated exposure reinforces the "pick a letter correctly" multiple-choice skill more directly than adding new data varieties does. Still beats d20 on every comparable metric.

Stop rate / long-form generation (repetition_penalty=1.15, not greedy)

Greedy decoding (temperature=0) is only the scoring protocol used above — it is not what this model is served with. repetition_penalty=1.15 was found empirically to be the deployment sweet spot (see this project's prepare/stoprate.py for the full derivation); nanochat's own interactive chat_cli.py also does not use greedy decoding by default.

English long-form success improved (68%→74%); Chinese is flat within noise (n=100 per language, standard error ~4-5 points). Not a clean sweep, but no regression either. "Long-form success" = >=150 tokens, repetition ratio <0.2, stopped naturally. Chinese open-ended creative writing (poems, stories) is still the weakest spot — it is more prone to repetition collapse than factual Q&A, a known, unresolved limitation (root cause not fully identified; greedy decoding is the primary suspect, not the training recipe, per this project's own ablations — see docs/POSTTRAIN.md).

The three targeted fixes from v9, re-checked after the second epoch

v9 fixed identity and translation but not one specific failure mode (responding to a remark rather than a question); the second epoch re-sees the same data, so this was re-tested rather than assumed fixed:

Repeating the same 8,000 rows of WildChat data a second time did not fix the remaining gap — as expected. This specific pattern (a bare compliment, no request attached) is naturally rare even in real conversation logs (WildChat mining found roughly 1% of candidate turns match it), so repetition reinforces already-thin signal rather than creating new signal. A synthetic, hand-written data source targeting this exact shape (analogous to how the Identity task itself is hand-written, not mined) is the planned next step, not included in this release.

What this model is / is not good at (from manual testing, not benchmarked)

This is a 524M-parameter model trained on 14.6B pretraining + 625M post-training tokens — for scale, Qwen3-0.6B-Base saw 36T tokens, about 2,700x more. Do not expect factual reliability. Manual testing surfaces confident-sounding but fabricated numbers on things like population figures, country counts, or basic arithmetic ("how many days in a week"). This tracks with what the letter-MC benchmark measures (does it follow instructions, pick a reasonable-sounding answer) rather than what it does not measure (is the answer actually correct) — the two are different questions, and this model was optimized against the former. Self-identification and translation requests are handled reliably; responding sensibly to a remark that isn't phrased as a question is still unreliable.

Tokenizer

The tokenizer is a compiled C++ extension, not loadable by AutoTokenizer. The release ships tokenizer.py and example_load.py:

pip install git+https://github.com/Ismantic/PieceTokenizer
python example_load.py

The model code (model.py, checkpoint.py) is bundled too — the package depends only on torch plus the tokenizer extension, not on transformers.

Training

Full pipeline, data mixes and every design decision (including the mistakes) are documented in https://github.com/Ismantic/Summer. Notably docs/WHY.md records why fp32 master weights are mandatory, why the learning rate schedule is WSD rather than cosine, and what the vocabulary swap cost.

License

Apache-2.0. Training corpora are public datasets (FineWeb-Edu, Cosmopedia, CCI3-HQ, SkyPile, WMT19, OPUS-100 and others; see data/source.py upstream). Please observe their respective licenses.