Downloads · 30 days
0
nazdef/gpt2small-en-it-nanochat-lr3e4-bs6-wsd-shortfastdecay8k-final5e6-webwiki-step4000
gpt2small-en-it-nanochat-lr3e4-bs6-wsd-shortfastdecay8k-final5e6-webwiki-step4000 is a text generation model from nazdef. Use it when you need the model to write or continue text. It is set up for custom. The card lists the license as other.
This repo stages step4000.pt from the local NanoChat EN/IT GPT-2-small-like WSD short-fast-decay web/wiki run 20260605fresh-gpt2small-lr3e4-bs6-wsd-shortfastdecay8k-final5e6-webwiki.
Downloads · 30 days
0
Access
Public
Updated Jun 6, 2026
Repo size
2.2 GB
Likes
0
Public
Click a slice to open those files.
.pt1.6 GB · 75%
From the Hugging Face model README
This repo stages step_4000.pt from the local NanoChat EN/IT GPT-2-small-like WSD short-fast-decay web/wiki run 20260605_fresh-gpt2small-lr3e4-bs6-wsd-shortfastdecay8k-final5e6-webwiki.
step_4000.pt~0.96Bval_loss_mixed80003.882301174948.535776128Important caveat: this run split into two different winners:
step_8000 won the run's internal online validationstep_4000 won the external repo-native benchmark used for checkpoint selectionOperationally, that means the run kept improving on its own internal validation while moving away from the best comparable checkpoint on the benchmark we actually use to rank releases.
Repo-native benchmark suite: configs/eval/20260521_pretrain_minimal_en_it_webwiki_step11000.yaml
Release checkpoint metrics:
val_loss_mixed: 5.1440ppl_mixed: 171.3970val_loss_en: 4.7796ppl_en: 119.0571val_loss_it: 4.0479ppl_it: 57.2766loop_rate: 0.725repeated_4gram_rate: 0.900distinct_2: 0.4251cloze_en_contains: 0.00cloze_it_contains: 0.08Ranking inside the checked saved checkpoints from this run:
step_4000 -> mixed=5.1440step_7000 -> mixed=5.3313step_5000 -> mixed=5.3651step_8000 -> mixed=5.3930step_6000 -> mixed=5.5364This checkpoint takes the current comparable crown in the tracked public slice:
20260530_fresh-gpt2small-lr2e4-bs6-wsd-earlydecay7000-final5e6-webwiki step_7000val_loss_mixed = 5.2158val_loss_mixed = 5.1440-0.0718Operational reading:
step_4000 is the best checkpoint of the lr3e4 / 8k run under the repo-native benchmarkstep_8000 is less loopy and more diverse on surface metrics, but loses clearly on the primary benchmark metricSo this release is a benchmark-first winner, not a claim that free-form generation quality is suddenly solved.
source_loss_books_en: 4.9947source_loss_books_it: 5.0274source_loss_code: 8.6152source_loss_web_en: 6.1538source_loss_web_it: 6.0197source_loss_wiki_en: 3.8654source_loss_wiki_it: 3.5642The capital of Italy is -> Rome: rank=2708, prob=0.0000197A small language model should -> be: rank=1, prob=0.34375La capitale d'Italia è -> Roma: rank=559, prob=0.0001488Un piccolo modello linguistico dovrebbe -> essere: rank=1, prob=0.1904297These probes are directional evidence only. The main selection rule here remains the repo-native benchmark result and the comparable cross-run ranking context above.
training_config.yamltokenizer.json + tokenizer_meta.json/mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500Mbffb58ef99b4bb27ea6772f5853c16d43607e4eb~0.96Bstep_4000.ptstep_4000.safetensorsstep_4000.safetensors.jsontraining_config.yamltokenizer.jsontokenizer_meta.jsonbest_validation.jsoneval_summary.jsoncomparison.jsonbenchmark_report.mdbenchmark_metrics.jsonbenchmark_scores.jsonbenchmark_source_losses.jsonprobe_step_4000_summary.jsoneval_metrics.jsonlmetrics.jsonlprobe_generations.jsonl2026-06-07_shortfastdecay8k_release_step4000.mdThis project uses a custom NanoChat inference/training stack. The easiest local UI in the source repo is the Chainlit checkpoint tester documented in the repo README.