Downloads · 30 days
50
42% of all-time downloads
nazdef/gpt2medium-en-it-nanochat-gpt2preln-lr2e4-bs2-ga48-wsd-anchor20k-final2e5-webwiki-step13500
gpt2medium-en-it-nanochat-gpt2preln-lr2e4-bs2-ga48-wsd-anchor20k-final2e5-webwiki-step13500 is a text generation model from nazdef. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as cc-by-sa-4.0.
This repository publishes the best scalar checkpoint from the medium GPT2PreLN anchor20k run evaluated in the full repo-native sweep currently recorded in this workspace.
Downloads · 30 days
50
42% of all-time downloads
All-time downloads
118
Public
Parameters
338M
6.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.pt4.1 GB · 59%
From the Hugging Face model README
step_13500This repository publishes the best scalar checkpoint from the medium GPT2PreLN anchor20k run evaluated in the full repo-native sweep currently recorded in this workspace.
This is an ordinary checkpoint release, not the definitive public-facing family base release.
Released checkpoint summary:
step_13500.ptstable-recipe-gpt2medium-gpt2preln-k20-wsd-lr2e-4-anchor20k-final2e5-webwiki2500 tokensarchitecture: gpt2, block_type: gpt2_prelayernorm337,671,424337,639,424RTX 4060 Ti 16GB) for training, CPU benchmarked selectionThis is a base model, not an instruction-tuned chat model.
The full checkpoint sweep across the evaluated medium run selected step_13500 as the best checkpoint on the main benchmark metric:
step_13500: val_loss_mixed = 4.4652step_14500: val_loss_mixed = 4.5796step_9500: val_loss_mixed = 4.5982step_12500: val_loss_mixed = 4.6220step_13000: val_loss_mixed = 4.6226So this release is not “latest wins because timestamp”. It is the current benchmark-selected medium winner for this run family.
The scalar winner is step_13500, but the behavior shortlist is:
step_13500step_14000step_14500Why step_14000 remained interesting:
loop_rate = 0.250distinct_2 = 0.6081repeated_4gram_rate = 0.725language_consistency_it = 0.875Why step_14500 is a good compromise:
val_loss_mixed = 4.5796distinct_2 = 0.5694repeated_4gram_rate = 0.750language_consistency_it = 0.850Practical reading:
step_13500 is the best scalar checkpointstep_14000 is one of the cleanest checkpoints on repetition/behavior proxiesstep_14500 trades a little scalar loss for a still-healthy qualitative profilestep_13500val_loss_mixed = 4.4652val_loss_en = 4.3219val_loss_it = 3.5930ppl_mixed = 86.9371ppl_en = 75.3304ppl_it = 36.3446Behavior snapshot:
loop_rate = 0.325distinct_2 = 0.5268repeated_4gram_rate = 0.800language_consistency_en = 0.975language_consistency_it = 0.850cloze_en_contains = 0.16cloze_it_contains = 0.30Source losses:
books_en = 4.4590books_it = 4.5447code = 7.2659web_en = 5.9069web_it = 5.4595wiki_en = 3.0141wiki_it = 3.0585Honest short read:
This model was trained on the bilingual EN/IT web + wiki dataset:
202605141153_fineweb50_wiki50_50en_50it_score100_2500context_5Btokens_tok_20260515_en50it50_webwiki_stratified_500M2500 tokens2500source_balanced0.05Main source groups:
epfml/FineWeb-HQ)epfml/FineWeb2-HQ)google/wiki40b)google/wiki40b)Training math:
2500248239,904So step_13500 saw approximately:
3.238704B tokens totalK = 9.5913 tokens per parameter relative to the native training-time parameter countThis release bundle includes:
step_13500.ptstep_13500.safetensorsmodel.safetensorsconfig.jsontraining_config.yamlbest_validation.json, metrics.jsonl, eval_metrics.jsonl, probe_generations.jsonl)eval_summary.json, comparison.json, benchmark_report.md, benchmark_metrics.json, benchmark_scores.json, benchmark_source_losses.json)from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
repo_id = "nazdef/gpt2medium-en-it-nanochat-gpt2preln-lr2e4-bs2-ga48-wsd-anchor20k-final2e5-webwiki-step13500"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
prompt = "La capitale d'Italia è"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)
outputs = model.generate(
input_ids=input_ids,
attention_mask=attention_mask,
do_sample=True,
max_new_tokens=64,
temperature=0.8,
top_k=50,
top_p=0.95,
repetition_penalty=1.1,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
This release is published with CC-BY-SA-4.0 as the practical downstream posture for the mixed training corpus used here.
The training mix includes:
Downstream users are responsible for checking whether their use, redistribution, or derivative packaging remains compatible with the obligations of the upstream datasets and their terms.