Downloads · 30 days
0
nazdef/gpt2small-en-it-nanochat-lr2e4-bs6-wsd-earlydecay3500-final5e6-webwiki-step7000
gpt2small-en-it-nanochat-lr2e4-bs6-wsd-earlydecay3500-final5e6-webwiki-step7000 is a text generation model from nazdef. Use it when you need the model to write or continue text. It is set up for custom. The card lists the license as other.
This repo stages the benchmark-selected checkpoint from the local NanoChat EN/IT GPT-2-small-like WSD early-decay-3500 web/wiki run 20260530fresh-gpt2small-lr2e4-bs6-wsd-earlydecay3500-final5e6-webwiki.
Downloads · 30 days
0
Access
Public
Updated Jun 3, 2026
Repo size
4.4 GB
Likes
0
Public
Click a slice to open those files.
.pt1.6 GB · 74%
From the Hugging Face model README
This repo stages the benchmark-selected checkpoint from the local NanoChat EN/IT GPT-2-small-like WSD early-decay-3500 web/wiki run 20260530_fresh-gpt2small-lr2e4-bs6-wsd-earlydecay3500-final5e6-webwiki.
step_7000.ptThis run's best online validation checkpoint was not the same as its best benchmark checkpoint.
step_16000validation_loss=3.8532891550validation_perplexity=47.1478851784step_7000This release follows the benchmark winner, not the online-validation winner.
Repo-native benchmark suite: configs/eval/20260521_pretrain_minimal_en_it_webwiki_step11000.yaml
Winner metrics:
val_loss_mixed: 5.2358ppl_mixed: 187.8785val_loss_en: 5.1587ppl_en: 173.9426val_loss_it: 4.0506ppl_it: 57.4308loop_rate: 0.675repeated_4gram_rate: 0.95cloze_en_contains: 0.02cloze_it_contains: 0.10cloze_en_exact: 0.00cloze_it_exact: 0.00Benchmark ranking across the closeout candidates from this run:
step_7000
mixed=5.2358en=5.1587it=4.0506step_16000
mixed=5.4730en=5.2994it=4.6116step_28000
mixed=5.6045en=5.3080it=4.7532step_29000
mixed=5.7228en=5.2835it=4.7360step_23000
mixed=5.9788en=5.4895it=4.8834source_loss_books_en: 4.7953source_loss_books_it: 4.8795source_loss_code: 8.2308source_loss_web_en: 5.8867source_loss_web_it: 6.0990source_loss_wiki_en: 4.0703source_loss_wiki_it: 3.9486The capital of Italy is -> Rome: rank=69, prob=0.0015792847A small language model should -> be: rank=1, prob=0.5La capitale d'Italia è -> Roma: rank=273, prob=0.0003337860Un piccolo modello linguistico dovrebbe -> essere: rank=1, prob=0.4785156250Factual probes remain weak in both languages, while the procedural prompts are strong next-token continuations. These probes are directional evidence only. The main selection rule here is the repo-native benchmark result.
Token estimate formula:
tokens_seen ~= step * batch_size * grad_accum_steps * sequence_length6 * 16 * 2500 = 240000 tokens per stepCurrent comparable leaderboard by the same primary metric val_loss_mixed:
earlydecay7000 step_7000
mixed=5.2158~1.68Bearlydecay3500 step_7000
mixed=5.2358~1.68Bgpt2small-en-it-nanochat-lr2e4-bs6-cosine-webwiki-step7000
mixed=5.3558~1.68Bgpt2small-en-it-nanochat-lr2e4-bs6-wsd-fastdecay-webwiki-step11000
mixed=5.3576~2.64Bgpt2small-en-it-nanochat-lr1e4-bs6-cosine-webwiki-step14000
mixed=5.4493~3.36BSo this release is still a meaningful checkpoint in the comparable public slice, but it does not beat the already published earlydecay7000 step_7000.
training_config.yamltokenizer.json + tokenizer_meta.json/mnt/apps/llm-nanochat/datasets/202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500Mstep_7000.ptstep_7000.safetensorsstep_7000.safetensors.jsontraining_config.yamltokenizer.jsontokenizer_meta.jsonbest_validation.jsoneval_summary.jsoncomparison.jsonbenchmark_report.mdbenchmark_metrics.jsonbenchmark_scores.jsonbenchmark_source_losses.jsonprobe_step7000_summary.jsoneval_metrics.jsonl, metrics.jsonl, probe_generations.jsonl2026-06-03_wsd_earlydecay3500_release_step7000.mdearlydecay3500 run family, not the top checkpoint of the broader comparable leaderboard