1gpu-llm-medium-en-it-base-v2

1gpu-llm-medium-v2

Official medium bilingual base model of the 1gpu-llm family, trained from scratch and continued on a single NVIDIA GeForce RTX 4060 Ti 16GB.

This release is a base language model, not an instruction-tuned chat model.

Release summary

• Released checkpoint: step_34200
• Parent/reference checkpoint: step_34000 from the main CPT run
• Continuation: optimizer-preserving decay-only continuation, global steps 34000 → 35700; this release is the early scalar winner at 34200
• Architecture: GPT-2-style decoder-only Transformer with pre-layernorm blocks
• Repo-native configuration: architecture: gpt2, block_type: gpt2_prelayernorm, norm_order: preln
• Parameters: 337,639,424 (~337.6M)
• Context window and training sequence length: 2500 tokens
• Languages: English and Italian

Why step_34200

The checkpoint was selected by a controlled 1000-token decoding comparison of step_34000, step_34200, and step_34800, using the same tokenizer, seed, device, precision, prompts, and generation budget across four presets.

Evaluation winner:

• checkpoint: step_34200
• preset: creative
• tuning score: 3.1683
• holdout score: 3.8405
• EOS termination: 100% on tuning and holdout
• truncation at 1000 tokens: 0%
• strict loop rate: 0%
• mean length: 410.6 tuning / 494.8 holdout tokens
• distinct-2: 0.9531 tuning / 0.9536 holdout
• language switches: 0%

The model remains imperfect: long completions can show factual or semantic drift. The conservative alternative is the parent step_34000 with the anti_loop preset, which is shorter and more controlled.

Official default generation settings

All official 1gpu-llm releases use the following short-response defaults in their published generation configuration and UI:

• preset: balanced
• max_new_tokens: 64
• temperature: 0.3
• top_k: 50
• top_p: 0.95
• repetition_penalty: 1.1
• no_repeat_ngram_size: 0
• early termination: model EOS only

These are the operational publication defaults. The 1000-token settings above belong to the checkpoint-selection evaluation and are intentionally separate.

Recommended generation

The following example uses the official balanced publication defaults:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo_id = "nazdef/1gpu-llm-medium-v2"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "La capitale d'Italia è"
prompt_ids = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
bos = torch.tensor([[tokenizer.bos_token_id]], dtype=prompt_ids["input_ids"].dtype)
input_ids = torch.cat([bos, prompt_ids["input_ids"]], dim=1)
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    outputs = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        do_sample=True,
        max_new_tokens=64,
        temperature=0.3,
        top_k=50,
        top_p=0.95,
        repetition_penalty=1.1,
        no_repeat_ngram_size=0,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For a more conservative response style, use anti_loop:

• temperature=0.3
• top_k=50
• top_p=0.9
• repetition_penalty=1.15
• no_repeat_ngram_size=4

Training provenance

The model was trained on a balanced English/Italian web + wiki corpus derived from:

• English FineWeb-HQ (epfml/FineWeb-HQ)
• Italian FineWeb2-HQ (epfml/FineWeb2-HQ)
• English and Italian Wiki40B (google/wiki40b)

The local packed dataset used a 2500-token sequence length and a 50/50 EN/IT source-balanced mix. The continuation loaded the model and AdamW optimizer state from the step_34000 checkpoint, deliberately ignored the saved scheduler state, and created a local wsd-decay-only scheduler with no warmup or rewarm.

Included provenance files:

• training_config.yaml
• decoding_grid_config.yaml
• decoding_grid_report.md
• export_manifest.json
• step_34200.safetensors.json

License

The model card uses CC BY-SA 4.0 as the release license. Training data comes from mixed upstream sources with their own terms, including FineWeb/FineWeb2 and Wiki40B. Downstream users are responsible for checking the applicable upstream dataset terms, attribution requirements, share-alike obligations, and any restrictions connected to the data or generated outputs in their intended use case.

Limitations

• This is a pretrained base model; it is not instruction-following aligned.
• It may hallucinate facts and drift semantically during long generations.
• English/Italian language consistency is strong in the selected decoding holdout, but it is not a guarantee for arbitrary prompts.
• The public checkpoint is an experimental single-GPU family release, not a claim of state-of-the-art performance.