qwen2-style-50k

qwen2-style-50k

A Qwen2-0.5B-architecture decoder LM (246M parameters) trained from scratch on CCI3-HQ Chinese text with the zh-pythia 50k tokenizer, using EleutherAI's gpt-neox. It is the architecture-controlled twin of the zh-pythia GPT-NeoX-style models: same data, tokenizer, and token budget — only the transformer architecture differs (Qwen2-style GQA/SwiGLU/RMSNorm vs GPT-NeoX-style). The goal is to test how much of the Chinese-benchmark behaviour difference between the two families is explained by architecture alone.

Training is complete: 24,414 steps = 102.4B tokens (~1 epoch of the 110.3B-token CCI3-HQ corpus, seq len 4096, ~4.19M tokens/step). Final training loss 3.02; held-out test loss 2.707 (perplexity 15.0). main holds the final checkpoint (step 24414); other snapshots are step<N> branches, Pythia-style: the early ladder 1–128 plus 5000, 7500, 10000, 14500, 15000, 17500, 20000.

Load

from transformers import AutoTokenizer, LlamaForCausalLM

tok = AutoTokenizer.from_pretrained("hai-hu/qwen2-style-50k")
model = LlamaForCausalLM.from_pretrained("hai-hu/qwen2-style-50k")
# earlier steps: add revision="step<N>"

Important: checkpoint format is Llama, not Qwen2 — on purpose

The model was trained in gpt-neox with biases on all four attention projections (Q/K/V and the output projection). HuggingFace's Qwen2ForCausalLM hard-codes o_proj bias=False, so a Qwen2-class export would have to drop the trained o-projection bias. On a validation batch at the final checkpoint that bias moves logits by up to 25 (mean 3.0) and flips ~78% of argmaxes — i.e. it is not negligible, so exporting as Qwen2 would silently produce a different, broken model.

LlamaForCausalLM with attention_bias=true (biases on q/k/v and o_proj) represents the trained model exactly; RMSNorm, SwiGLU, rotary embeddings, GQA and tied embeddings are otherwise identical between the two classes. Conversion was verified to logits parity against the original gpt-neox checkpoint: max |Δlogit| = 4e-5 (fp32 numerics), 100% argmax agreement.

So: load it with LlamaForCausalLM (as above) and treat it as a Qwen2-family architecture. The weights are untouched originals.

Architecture

Training hyperparameters

Adam(0.9, 0.95), lr 3e-4 → cosine → 3e-5, warmup 8.2% (2000 steps), weight decay 0.1, grad clip 1.0, bf16, ZeRO-1, global batch 1024 sequences × 4096 tokens. Data: CCI3-HQ (~110B-token corpus, ~1 epoch for 24414 steps). Checkpoint every 250 steps; per-step branches here are a subset.

Hardware: 2× NVIDIA RTX PRO 6000 Blackwell (96GB), ~37.4 s/step, ~12.5 days wall clock.