Downloads · 30 days
42
15% of all-time downloads
nazdef/gpt2small-en-it-nanochat-gpt2preln-cpt8600-lr5e5-predecay-step18000
gpt2small-en-it-nanochat-gpt2preln-cpt8600-lr5e5-predecay-step18000 is a text generation model from nazdef. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-sa-4.0.
This repository packages the current best continual-pretraining checkpoint before any decay-only extraction in the small GPT2PreLN EN/IT line.
Downloads · 30 days
42
15% of all-time downloads
All-time downloads
272
Public
Parameters
112M
2.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.pt1.3 GB · 57%
From the Hugging Face model README
step_18000This repository packages the current best continual-pretraining checkpoint before any decay-only extraction in the small GPT2PreLN EN/IT line.
Concretely, this is:
nazdef/1gpu-llm-small-en-it-basestep_8600.ptstep_18000.pt5e-5So this repo should be read as:
a public CPT experiment checkpoint derived from
1gpu-llm-small-en-it-base, not yet the final post-decay family-base promotion.
1gpu-llm-small-en-it-base was the previous public small-family winner, produced by a short decay-only continuation up to step_8600.
This repo starts from that released checkpoint and continues pretraining with a lower learning rate and a new local schedule:
step_8600.ptcontinual_pretraining5e-5500100001000The resulting branch was evaluated checkpoint-by-checkpoint on CPU, and step_18000 emerged as the best checkpoint of the branch.
gpt2_prelayernorm2500320007681212136,128,000 (~136.128M)This is a base model, not an instruction-tuned chat model.
nazdef/1gpu-llm-small-en-it-basestep_8600.pt20260622_resume-gpt2small-gpt2preln-k20-wsds800-final2e5-webwiki-step8000-dense5020260701_continual-pretraining-gpt2small-step8600-lr5e5-w500-s10000-d1000-final1e5-webwikistep_18000.ptImportant release caveat:
wsd-decay-only extraction stage yetstep_18000 is therefore the best pre-decay continual-pretraining checkpoint, not the final cooled-down release candidateThis branch stayed on the same bilingual EN/IT web+wiki family used by the small base release:
202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500Msource_balanced25000.05Main source groups:
epfml/FineWeb-HQ)epfml/FineWeb2-HQ)google/wiki40b)google/wiki40b)Training math:
2500616240,000Approximate token exposure:
step_8600: about 2.064B tokens8600 -> 18000: about 2.256B tokensstep_18000: about 4.320B tokensstep_18000 Was ChosenThe branch was evaluated in four benchmark passes:
12000/12500 quick batchstep_20000Lower val_loss_mixed is better.
Key checkpoints:
step_8600: 4.7964step_9500: 4.7565step_12500: 4.8020step_18000: 4.7358step_20000: 4.7892So:
step_180001gpu-llm-small-en-it-base1gpu-llm-small-en-it-base corresponds to the source step_8600 checkpoint.
| checkpoint | val_loss_en | val_loss_it | val_loss_mixed | ppl_mixed | cloze_it_contains | language_consistency_it | repeated_4gram_rate | loop_rate |
|---|---|---|---|---|---|---|---|---|
step_8600 | 4.8075 | 3.6415 | 4.7964 | 121.0694 | 0.20 | 0.825 | 0.925 | 0.525 |
step_18000 | 4.7260 | 3.8244 | 4.7358 | 113.9571 | 0.28 | 0.775 | 0.875 | 0.450 |
Where step_18000 is better:
val_loss_mixed: 4.7964 -> 4.7358val_loss_en: 4.8075 -> 4.7260ppl_mixed: 121.07 -> 113.96loop_rate: 0.525 -> 0.450repeated_4gram_rate: 0.925 -> 0.875cloze_it_contains: 0.20 -> 0.28Where step_8600 is still better:
val_loss_it: 3.6415 vs 3.8244language_consistency_it: 0.825 vs 0.775Short honest read:
step_18000The capital of Italy is -> expected Rome
correct_token_rank = 7correct_token_probability = 0.0172perplexity_on_target_sequence = 58.10A small language model should -> expected be
correct_token_rank = 1correct_token_probability = 0.5547perplexity_on_target_sequence = 1.80La capitale d'Italia è -> expected Roma
correct_token_rank = 5correct_token_probability = 0.0356perplexity_on_target_sequence = 28.05Un piccolo modello linguistico dovrebbe -> expected essere
correct_token_rank = 1correct_token_probability = 0.3320perplexity_on_target_sequence = 3.01Practical reading:
No dedicated decoding sweep has been run yet on step_18000.
For now, this repo ships the same conservative public preset used by nazdef/1gpu-llm-small-en-it-base:
balanceddo_sample = truetemperature = 0.8top_k = 50top_p = 0.95repetition_penalty = 1.1no_repeat_ngram_size = 0max_new_tokens = 64This is an inherited safe default, not a claim that step_18000 already has a dedicated decoding-search winner.
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
repo_id = "nazdef/gpt2small-en-it-nanochat-gpt2preln-cpt8600-lr5e5-predecay-step18000"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
prompt = "A small language model should"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)
outputs = model.generate(
input_ids=input_ids,
attention_mask=attention_mask,
do_sample=True,
max_new_tokens=64,
temperature=0.8,
top_k=50,
top_p=0.95,
repetition_penalty=1.1,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
.pt checkpoint.safetensors weights plus metadata sidecarmodel.safetensorsconfig.jsonbest_validation.jsonmetrics.jsonleval_metrics.jsonlprobe_generations.jsonlstep_18000generation_config.jsonrecommended_decoding_params.jsonrelease_note.mdUse this model as:
1gpu-llm-small-en-it-basewsd-decay-only cooldown tailDo not read this repo as:
1gpu-llm-small-en-it-baseThat final decision belongs to the next decay-only stage.