Downloads · 30 days
0
bishnoiyash/Qwen3-0.6B-reproduction
Qwen3-0.6B-reproduction is a text generation model from bishnoiyash. Use it when you need the model to write or continue text. It is set up for pytorch. The card lists the license as apache-2.0.
A single-file PyTorch reimplementation of Qwen/Qwen3-0.6B-Base, written from a blank file and checked against the official Hugging Face weights.
Downloads · 30 days
0
Access
Public
Updated Aug 5, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.md16.2 KB · 91%
From the Hugging Face model README
A single-file PyTorch reimplementation of Qwen/Qwen3-0.6B-Base, written from a blank file and checked against the official Hugging Face weights.
Parity result: max absolute logit error 0.0 and relative error 0.0 against the official weights, argmax token 12095 which decodes to " Paris", on a 5-token prompt, input shape [1, 5], fp32 on CPU, tolerance gate 1e-3. Recorded in results/verify.json.
Scope of that claim, stated plainly: it is a CPU fp32 check on one short prompt. This repository has no GPU parity run, no long-context parity run and no per-layer parity run for Qwen3. It should not be read as bit-exactness under bf16, under CUDA kernels, or at length.
Parameter count 596,049,920, measured at run time and stamped in run_sft_seed0.log line 7. An earlier version of this card sourced it to verify.json, where it does not appear.
Weights: none published, and none exist to publish. verify.py downloads the official weights at run time and saves nothing, so no checkpoint in this project is "the parity-verified reproduction". The training checkpoints described below are separate artifacts. Code, logs and results: https://github.com/yashb98/BuildFromScratch
The corrected READMEs are on the default branch. PR #5 merged on 2026-08-05, fast-forwarding main to 6911743, so the repository and this page agree.
Note on metadata: the Hugging Face model tree lists this repository as a finetune of Qwen/Qwen3-0.6B-Base. That is the closest relation the schema allows, but it is not accurate. This is an independent reimplementation of the architecture, checked against the official weights as a correctness oracle. It is not derived from them by further training.
RMSNorm, rotary embeddings with theta 1e6, grouped-query attention, SwiGLU, and Qwen3's per-head QK-Norm applied before RoPE. Get any one of them subtly wrong and the model still loads and still runs. The parity gate is what proves it does not.
One known config divergence: model.py defaults max_position_embeddings to 40,960 while the cached Qwen3-0.6B-Base config.json says 32,768. Thirteen of the fourteen defaults match; that one does not. It is harmless for weight loading because the RoPE buffers are non-persistent, but 40,960 is not the released model's context length and should not be quoted as such.
The three perplexities below were not all measured on the same validation slice, so they do not form one comparison.
| Run | Training tokens | FineWeb-Edu val PPL | Val slice |
|---|---|---|---|
| Phase A, 2,000 steps | 131,072,000 | 46.31 | A (val sha1 8ad9e246b0bf63bd) |
| Released Qwen3-0.6B-Base | ~36T, per tech report | 13.40 | A (val sha1 8ad9e246b0bf63bd) |
| Phase B faithful baseline, AdamW + cosine | 1,189,478,400 | 28.65 | B (val sha1 ad3513719d0f81e4) |
Only the first two rows are like-for-like: 46.31 / 13.40 = 3.456x, same slice, same eval code. The 28.65 figure was measured against a different 300,000-token slice, so 28.65 / 13.40 = 2.14x is a cross-slice ratio and is not published here as a result. An earlier version of this card, and the repository README, claimed an "identical 300k-token FineWeb-Edu val slice" for that pair. That claim was false and is retracted.
The 13.40 figure was measured by this repository's own eval code on 2026-06-09. The ~36T token count is copied from the Qwen3 technical report (arXiv:2505.09388), not measured here.
Data-scale gap: 36T against the 1.19B-token run is roughly 30,000x. Against the 131M-token Phase A run it is roughly 275,000x. An earlier version of this card attached the 275,000x figure to the 1.19B run. That was wrong.
How much of this is slice choice: the same 1.19B-token checkpoint that scores 28.65 on its own slice scores 24.55 on a held-out FineWeb-Edu split from the dataset-forge pipeline, 202 documents and 204,600 scored tokens at sequence length 1024 and stride 512. A 14.3 percent swing from slice and windowing alone, which is larger than several effects reported at this scale.
Validation contamination: the Phase-B validation slice is a sequential continuation of the training stream and was not decontaminated. Document-disjoint splitting and 13-gram decontamination landed in commit 86e79f3 on 2026-06-16, after every Phase-B run had finished, and the token caches those runs consumed do not carry the decontam key. Treat the Phase-B perplexities as leak-suspect.
n = 1: every perplexity above is one run at one seed with no confidence interval. This project's own governance rules name single-seed FineWeb val perplexity as a metric that must not carry a headline on its own, and that applies to this card.
Corpus HuggingFaceFW/fineweb-edu, config sample-10BT, split train, streamed. Validation slice 300,000 tokens. Sequence length 4,096 with 50 windows at stride equal to sequence length, so windows do not overlap, giving 204,800 scored tokens. Tokenizer Qwen/Qwen3-0.6B-Base. bf16 forward, cross-entropy accumulated in fp32 in chunks of 8,192.
This is not wikitext and it is not comparable to published wikitext perplexity numbers.
Vocabulary: 151,936 is the model config's vocab_size and the width of the logit vector. The tokenizer itself has 151,669 entries.
Run 2026-06-16 on checkpoint_qwen3_baseline2tpp.pt at sequence length 1024, stride 512, 200 windows, so these windows do overlap and positions are counted more than once. Dataset revisions pinned.
| Task | PPL | BPB | Scored tokens |
|---|---|---|---|
| wikitext-2-raw-v1 validation | 37.01 | 1.2256 | 204,600 |
| codeparrot-clean validation | 438.67 | 2.1286 | 204,600 |
This suite was never run against the released Qwen3-0.6B-Base, so there is no suite-comparable counterpart to 13.40.
n = 500 per task with Wilson intervals, 2026-06-24. Mean BPB-gold 1.188.
| Task | Faithful baseline | Reading |
|---|---|---|
| LAMBADA accuracy | 0.170, CI 0.140 to 0.205 | |
| ARC-Easy acc_norm | 0.454 | above the 0.25 chance line |
| HellaSwag acc_norm | 0.348 | above chance, but only marginally and with a wide interval; not headline-bearing |
| Winogrande | 0.500 | exactly chance; no signal |
Phase B faithful baseline: 18,150 steps at 65,536 tokens per step, 1,189,478,400 tokens. Sequence length 4,096, micro-batch 4, gradient accumulation 4, so 16 sequences per step. Full bf16 with no autocast, no GradScaler and no fp32 master weights. AdamW with betas 0.9 and 0.95 and eps 1e-8, peak learning rate 2.4e-3 on a cosine down to 3.2e-4, warmup 900 steps, weight decay 0.01 applied only to parameters of rank 2 or higher, gradient clipping 1.0, seed 0, torch.compile enabled, CUDA memory fraction 0.85.
Wall clock 2,663.1 minutes, about 44.4 hours, plus about 26 minutes to stream and tokenize the corpus. Throughput 7,444 tokens per second as the cumulative average at the end of the run; the 7,480 figure quoted elsewhere is the reading at step 100. Peak memory 52.4 GB. torch.compile was worth 1.89x, 7,167.4 against 3,787.5 tokens per second.
Provenance of that recipe: it is not reconstructed from prose. The full-state checkpoint carries a training_recipe dictionary written from the run's own arguments, naming the recipe A_faithful_scaled_cosine and recording seq_len 4096, micro_batch 4, grad_accum 4, token_budget 1189478400, total_steps 18150, peak_lr 0.0024, end_lr 0.00032, warmup_steps 900, weight_decay 0.01, grad_clip 1.0, betas 0.9 and 0.95, eps 1e-8, seed 0, dtype bfloat16 and compile true. The same file records trained_ppl 28.650247632362024 and a random-initialisation baseline_ppl of 185,810.49, which is about the size of the vocabulary, as it should be.
Original finding: 2.4e-3 was the best learning rate found at this scale, and Qwen never published the 0.6B configuration. At the 131M budget the siblings were 46.89 at 1.7e-3 and 49.28 at 3.0e-3 against 46.31 at 2.4e-3.
Under-training: at roughly 2 tokens per parameter this run sits about 10x below the Chinchilla-optimal budget for 596M parameters, which is near 12B tokens. It is not trained to the released model's quality and is not a substitute for it.
At the smallest budget the effect is large. Three seeds per arm, 640 steps of 65,536 tokens for 41,943,040 tokens per cell, NorMuon at lr 0.011 on 196 two-dimensional parameters and AdamW at 2.4e-3 with weight decay 0 on the remaining 114. Wikitext BPB: AdamW mean 2.1098, NorMuon mean 1.6355, difference -0.4743, 95 percent CI 0.443 to 0.505.
Four things qualify that interval. Degrees of freedom were 3.86 by Welch-Satterthwaite but the interval was computed with df floored to 3 and t 3.182. The test is an unpaired Welch test on a design that is paired by seed. The data split seed was fixed at 0 across all six cells, so the standard error understates the true variance. And both arms ran two-dimensional weight decay at 0.1, which is NorMuon's tuned value, while AdamW's faithful recipe was tuned at 0.01, so the baseline ran at ten times the weight decay it was tuned for and some of the gap may be a handicap rather than an effect.
Now sweep the budget at fixed parameter count.
| Token budget | Wikitext BPB gap | Code BPB gap |
|---|---|---|
| 42M | 0.4743 | 0.5016 |
| 168M | 0.1259 | 0.1758 |
| 420M | 0.0717 | 0.1771 |
On wikitext the gap converges toward zero and the project ledger records the verdict as null. On the code corpus the gap stops falling between 168M and 420M and edges up, which is a plateau rather than convergence; only the three-point regression slope is negative, at -0.342 with r-squared 0.841. Learning rates were not re-tuned per horizon.
One detail that looks like an error and is not: the ablation cells ran a cosine schedule with 50 warmup steps, which is the right order of magnitude for a 640-step cell. The 900-step warmup belongs to the 18,150-step Phase-B runs. The two numbers describe different experiments and should not be reconciled.
The honest summary: -0.474 BPB is a small-budget artifact. At 420M tokens the wikitext gap is 0.0717 with a 95 percent interval of 0.0553 to 0.0881 over three seeds, which is statistically significant and practically small. Anyone quoting -0.474 without this ladder is quoting the wrong number. The root repository README carried an earlier two-seed version of this ladder; as of 2026-08-05 it carries the same three-seed ladder and verdict as this card.
Filenames containing 42M, 168M and 420M refer to token budgets, not parameter counts.
Parity, and this one works from a clean checkout:
python verify_run.py
It writes results/verify.json and exits 0 or 1.
Perplexity:
python eval_original_vs_repro.py
This does not run from a fresh clone. It hard-codes CUDA with no CPU fallback, and it needs a 1,066,978,101-byte token cache plus 3,576,719,229 bytes of checkpoints, all gitignored.
Environment: there is no requirements.txt, no pyproject.toml and no lockfile in this project, and the 13.40 evaluation did not stamp its library versions. Install is an unpinned one-liner. There are no git tags and no result file carries a commit hash, so these artifacts cannot be pinned to a revision from the outside.
The project ledger holds an eval entry for this checkpoint at research/ledger/runs/2026-06-16_qwen3-faithful_eval-first.md, typed as an eval and carrying the suite numbers above. No ledger entry records the training run itself. An earlier version of this card said no ledger entry existed at all, which was wrong.
Raw torch.save pickles, not safetensors. The project holds 89 model checkpoints totalling about 216 GB, plus 26 token caches totalling about 23 GB. The single largest .pt file in the repository is a token cache, not a model.
Full-state checkpoints are about 3.577 GB and carry the keys model, config, step, tok_seen, baseline_ppl, trained_ppl, training_recipe, optim, sched, rng_torch, rng_cuda, rng_numpy and rng_python. They fail torch.load with weights_only=True, because restoring the numpy RNG state needs an unpickler global. The state dict has 311 tensors, no torch.compile prefix, all bf16, 751,632,384 elements including the tied duplicate and 596,049,920 unique. Weights-only ablation cells are about 1.192 GB, IMU-1 step checkpoints about 1.193 GB, and the architecture-flag variants about 3.579 GB, the extra 458,752 parameters being the head-gating weights.
An earlier version of this card listed the checkpoint keys as arm, seed, fineweb_val_ppl and recipe. That list was wrong; the keys above are what the training script actually writes.
There is no config.json anywhere in the repository, no vendored tokenizer files, and no safetensors converter for Qwen3. Of the Qwen3 arms the faithful baseline is the only clean publish candidate: stock architecture, no compile prefix, and a complete recipe. The IMU-1 and partial-RoPE checkpoints would load as a different architecture than advertised unless a custom modeling file shipped alongside them.
Loader: Qwen3ForCausalLM(cfg: Qwen3Config). There is no from_pretrained and no init.py, so imports are flat and need the working directory or sys.path set. forward(input_ids, labels=None, attention_mask=None) returns a plain dict with logits and loss. generate(input_ids, max_new_tokens=64, temperature=0.8, top_k=50) has no KV cache. Weight loading is load_state_dict(strict=False) with only lm_head.weight permitted to be missing, because it is tied.
Single machine, single GPU, self-directed research. This is an architecture reproduction plus controlled experiments, not a production or distributed-scale training run. Parity is CPU fp32 only. Training perplexities are single-seed on a validation slice that was not decontaminated at the time. It is not trained to the released model's quality and is not a drop-in replacement for it.
Part of the BuildFromScratch collection.
Modernized training recipe built on this baseline: https://huggingface.co/bishnoiyash/Qwen3-0.6B-imu1
Smaller verified reproduction, a good starting point: https://huggingface.co/bishnoiyash/SmolLM2-135M-reproduction
@misc{bishnoi_qwen3_reproduction,
author = {Bishnoi, Yash},
title = {Qwen3-0.6B reproduction (logit parity, CPU fp32)},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/bishnoiyash/Qwen3-0.6B-reproduction}
}
This model was pre-trained on FineWeb-Edu
(sample-10BT), which is released under the
Open Data Commons Attribution License (ODC-By) v1.0
and is additionally subject to CommonCrawl's Terms of Use.
ODC-By section 4.3 requires this notice on any Produced Work, which a model trained on the database is:
Contains information from FineWeb-Edu, which is made available under the ODC Attribution License.
@misc{lozhkov2024fineweb-edu,
author = {Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas},
title = {FineWeb-Edu: the Finest Collection of Educational Content},
year = {2024},
url = {https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu},
doi = {10.57967/hf/2497},
publisher = {Hugging Face}
}
The training loader (train_qwen3.py:151) streams the dataset without pinning a
revision, so the exact snapshot used is not recoverable from the code.
Yash Bishnoi, Edinburgh, UK.
GitHub: https://github.com/yashb98 Portfolio: https://yashbishnoi.io LinkedIn: https://www.linkedin.com/in/yash-bishnoi/