ode-pessoana

ode · Pessoana char-GPT

A small (~10.7M-parameter) character-level GPT, trained from scratch in tinygrad on the Pessoana corpus of Portuguese classic literature. It writes Portuguese prose and verse one character at a time.

Model details

• Architecture: decoder-only transformer (GPT) — 6 layers, 6 heads, 384-dim, context 256, weight-tied embeddings. ~10.7M parameters.
• Tokenizer: character-level, 162-character vocabulary (meta.pkl).
• Framework: tinygrad (the whole model is ~130 lines, in model.py).
• Training: 5000 steps on an NVIDIA A100. AdamW (weight decay 0.1 on 2-D weights only), cosine LR 1e-3 → 1e-4 with 100-step warmup, gradient clipping 1.0, dropout 0.2, batch size 64, block size 256.
• Best validation loss: 1.361 nats (1.96 bits/char).

Usage

The model ships with its architecture (model.py) and a sampler (sample.py):

pip install tinygrad numpy huggingface_hub
huggingface-cli download vreabernardo/ode-pessoana --local-dir ode-pessoana
cd ode-pessoana
python sample.py --prompt "A cidade" --tokens 500 --temperature 0.8

model.safetensors holds the weights; config.json the dimensions; meta.pkl the character vocabulary for encode/decode.

Sample output

Prompt "A cidade":

A cidade da camisa me faz ali perdão; é para me lembrar o que me diz em que casada,
se me engendrara. Eu não custaria a minha graça, a ser só a sua...»

A RAINHA
— Mas é assim. De mim não te amendo a distância, com filha, as manas, se opusesse de si.

D. LANCEROTE
— Se pensar o meu quarto, que agradecesse o rei...

D. TIBÚRCIO
— Continuou ela. E eu não o quis que me disse que eu fosse a torre a catarra.

Training data

Pessoana — 84 public-domain works of Portuguese and Brazilian classic literature (Eça de Queirós, Machado de Assis, Camilo Castelo Branco, Florbela Espanca, Fernando Pessoa…), from Projecto Adamastor, under CC BY-SA 4.0.

Limitations

A character-level model of this size learns Portuguese spelling, morphology, dialogue formatting, and literary rhythm, but not factual or long-range semantic coherence — the output reads like Portuguese literature without meaning much. It uses pre-1990 orthography.

License

CC BY-SA 4.0, inheriting the ShareAlike terms of the training corpus.

Citation

@misc{vrea2026odepessoana,
  title     = {ode: A Character-Level GPT for Portuguese Literature},
  author    = {Vrea, Bernardo},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/vreabernardo/ode-pessoana}
}