20260713_resume-gpt2medium-step31000-d2000-step33000

GPT2Medium EN/IT NanoChat — 31k decay checkpoint step_33000

This is an ordinary, non-official checkpoint release from a decay-only continual-pretraining branch. It is a candidate for comparison in the future 1gpu-llm-medium-v2 selection; it is not the definitive v2 release.

Released checkpoint

• checkpoint: step_33000.pt
• branch: 20260713_resume-gpt2medium-gpt2preln-k20-wsddecayonly-cpt14700-step31000-lr5e5-final1e5-webwiki-d2000
• parent checkpoint: step_31000.pt
• schedule: 2000 decay steps, final target step_33000, final LR 1e-5
• languages: English + Italian
• context window: 2500 tokens
• architecture: GPT-2-style decoder with pre-layernorm blocks
• architecture identifiers: architecture: gpt2, block_type: gpt2_prelayernorm
• parameter count: 337,639,424 in the verified Transformers export
• hardware: single RTX 4060 Ti 16GB

Selection and position

step_33000 is the scalar winner of the 31k decay branch:

• val_loss_mixed = 4.4857
• mixed perplexity: 88.74
• val_loss_en = 4.4539
• val_loss_it = 3.5831
• local branch alternative step_32200: 4.4995

The endpoint is not automatically the best behavioral checkpoint. The branch has substantial repetition in generated samples. For comparison, the retained alternatives are step_31100 (cleanest repetition proxies) and step_31900 (best diversity proxies).

The current global scalar comparison remains:

• step_34000 no-decay CPT: 4.4401
• step_36000 d2000 from 34k: 4.4493
• step_23100 d1800 from 22k: 4.4675
• step_33000 d2000 from 31k: 4.4857

This checkpoint is therefore a branch winner and comparison candidate, not an official family release.

Training data

The model was trained on the bilingual EN/IT web + wiki corpus:

• English FineWeb-HQ (epfml/FineWeb-HQ)
• Italian FineWeb2-HQ (epfml/FineWeb2-HQ)
• English and Italian Wiki40B (google/wiki40b)
• local dataset: 202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo_id = "nazdef/20260713_resume-gpt2medium-step31000-d2000-step33000"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "La capitale d'Italia è"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)
outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_mask,
    do_sample=True,
    max_new_tokens=64,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

This is a base pretraining checkpoint, not an instruction-tuned chat model.

License

This release uses CC-BY-SA-4.0 as the practical downstream posture for the mixed training corpus. The corpus combines FineWeb-HQ/FineWeb2-HQ web data and Wiki40B slices, whose upstream terms and attribution/share-alike obligations may apply to downstream use and redistribution. Users are responsible for checking that their intended use and derivative packaging comply with the upstream dataset terms.