gpt2-ptbr-218m

gpt2-ptbr-218m

Portuguese GPT-2-like autoregressive language model trained from scratch.

Training pipeline

The released model, gpt2-ptbr-218m, is the final checkpoint of a three-stage pipeline:

1. Pretraining on Portuguese Wikipedia.
2. Supervised Fine-Tuning on Alpaca PT-BR.
3. Supervised Fine-Tuning on Canarim-Instruct-PTBR.

The released checkpoint corresponds to the final instruction-tuned model.

Model description

• Architecture: GPT-2 (decoder-only transformer)
• Parameters: 218,040,320 trainable unique parameters (weight-tying between token embedding and output projection; zero biases added for Hugging Face compatibility)
• Layers: 16
• Attention heads: 16
• Embedding dimension: 1024
• Vocabulary: 16,000 tokens (SentencePiece BPE)
• Sequence length: 256 tokens
• Activation: GELU
• Dropout: 0.1

Note on parameter count: During export, the weight-tying between the token embedding and the language model head is preserved, and zero-initialized bias tensors are added for compatibility with Hugging Face's GPT2LMHeadModel. These biases are not part of the original trained model and do not affect behavior.

Datasets

• Portuguese Wikipedia corpus — used for autoregressive pretraining.
• Alpaca PT-BR — Portuguese instruction-following dataset derived from Stanford Alpaca (Taori et al., 2023). Used for the first SFT stage.
• Canarim-Instruct-PTBR — Portuguese instruction-following dataset by Maicon Domingues (Domingues, 2023). Used for the second SFT stage.

Tokenizer

• Type: SentencePiece BPE
• Vocabulary: 16,000 tokens
• Special tokens: <unk> = 0, <bos> = 1, <eos> = 2, <pad> = 3
• Pre-tokenizer: Metaspace (SentencePiece native)
• Model max length: 256 tokens

The tokenizer was trained from scratch on the Portuguese Wikipedia corpus.

Training details

• Type: sft_response_only
• Best validation loss: 2.1538288593292236
• Training steps: 1050

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "augustoafleal/gpt2-ptbr-218m"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "A inteligência artificial é"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.7,
    top_k=40,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

• The model may generate factual errors.
• The model may repeat phrases.
• The model may fail to follow instructions exactly.
• The context length is limited to 256 tokens.
• The model was trained on limited data compared to modern LLMs.
• This model is not suitable for high-stakes use without human validation.

Dataset citations

Stanford Alpaca: Taori et al., 2023. https://github.com/tatsu-lab/stanford_alpaca
Canarim-Instruct-PTBR: Domingues, 2023. https://huggingface.co/datasets/dominguesm/Canarim-Instruct-PTBR

Citation

If you use this model, please cite:

@misc{gpt2ptbr218m,
  title        = {gpt2-ptbr-218m: A Portuguese GPT-2-like Autoregressive Language Model},
  author       = {Augusto Antônio Fontanive Leal},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/augustoafleal/gpt2-ptbr-218m}}
}