inacio-d16v1

inacio-d16v1

inacio-d16v1 is the first public base model of the nanochatpt project, an effort to train compact language models from scratch with European Portuguese (PT-PT) as the primary language while retaining English competence.

The model is based on Andrej Karpathy's NanoChat architecture and was pretrained from scratch on a multilingual corpus deliberately biased towards Portuguese.

This release is a base model, intended for further supervised fine-tuning and research rather than direct conversational use.

Model Architecture

• Architecture: NanoChat
• Decoder-only Transformer
• Parameters: 872,416,058
• Layers: 16
• Hidden size: 1024
• Attention heads: 8
• KV heads: 8
• Context length: 2048 tokens
• Vocabulary: 65,536 tokens
• Activation: ReLU²
• Attention: Multi-Query Attention (MQA)
• Positional encoding: RoPE
• Window pattern: SSSL

Pretraining Dataset

The model was trained on a custom corpus composed primarily of European Portuguese data.

Training data includes:

• DuarteoCarmo Bagaco (PT)
• Karpathy ClimbMix (EN)
• Europarl Portuguese
• Filtered PT-PT subset of CulturaX

The corpus intentionally mixes Portuguese 80% and English 20% to preserve multilingual capabilities while prioritizing PT-PT.

Training

Training was performed on the Modal cloud platform.

Hardware:

• 4 × NVIDIA A100 SXM4 80GB

Training configuration:

• Depth: 16
• Batch size: 524,288 tokens
• Target token/parameter ratio: 12
• Total training tokens: 3.22B
• Total optimization steps: 6,144

Training completed successfully without divergence.

Final metrics:

• Minimum validation BPB: 0.769
• CORE metric: 0.172

Training costs

• total 37$ - 30$ = 7$ (modal gives 30$ credit every months)

Intended Uses

This model is intended for:

• continued pretraining
• supervised fine-tuning
• research
• experimentation
• educational purposes

It is not intended to be used directly as a chat assistant.

Limitations

This is first complete training run of the nanochatpt project.

Although the model acquires a substantial amount of Portuguese linguistic knowledge during pretraining, it still exhibits important limitations typical of compact base models trained on relatively modest token budgets:

• factual inaccuracies
• arithmetic weaknesses
• reasoning limitations
• hallucinations
• incomplete world knowledge

These limitations are expected to improve with larger pretraining corpora and subsequent instruction tuning.

Requeriments

nanochatpt or nanochat cloned to your machine, install requirements. https://github.com/inaciose/nanochatpt

copy the model to your local folder copy the tokenizer to your local folder

cache/modal ├── base_checkpoints │   └── d16 │   ├── meta_0XXXXX.json │   ├── model_0XXXXX.pt │   ├── optim_0XXXXX_rank0.pt │   ├── optim_0XXXXX_rank1.pt │   ├── optim_0XXXXX_rank2.pt │   └── optim_0XXXXX_rank3.pt └── tokenizer ├── token_bytes.pt └── tokenizer.pkl

export NANOCHAT_BASE_DIR=cache/modal

Project

nanochatpt is an personal open research project focused on training compact language models from scratch with European Portuguese as the primary language.

Future releases may include:

• larger pretraining corpora
• improved PT-PT datasets
• Portuguese evaluation benchmarks
• improved instruction tuning