inacio-sft-d16v1

inacio-sft-d16v1

inacio-sft-d16v1 is the first instruction-tuned model of the nanochatpt project. It is obtained by supervised fine-tuning (SFT) of the base model inacio-d16v1, with the objective of improving conversational abilities while maintaining European Portuguese (PT-PT) as the primary language.

This release represents the first end-to-end demonstration of the complete nanochatpt training pipeline, from tokenizer training to pretraining and supervised fine-tuning.

Base Model

This model is fine-tuned from:

• inaciose/inacio-d16v1

The base model was pretrained from scratch on a corpus primarily composed of European Portuguese text, together with a smaller proportion of English.

Model Architecture

• Architecture: NanoChat
• Decoder-only Transformer
• Parameters: 872,416,058
• Layers: 16
• Hidden size: 1024
• Attention heads: 8
• KV heads: 8
• Context length: 2048 tokens
• Vocabulary: 65,536 tokens

Supervised Fine-Tuning Dataset

The instruction tuning stage uses NanoChat's TaskMixture framework.

Training data consists of:

• CAIS MMLU (English)
• OpenAI GSM8K (English)
• inaciose/smol-smoltalk-pt-v1 (Mix European Portuguese 80%, English 20%)

The Portuguese conversational dataset smol-smoltalk-pt-v1 was created specifically for this project as a temporary PT-PT replacement for HuggingFaceTB/smol-smoltalk.

It is derived from:

• AMALIA-LLM-0626-SFT-Dataset

Future releases will progressively replace the remaining English instruction datasets with Portuguese equivalents.

https://huggingface.co/datasets/cais/mmlu https://huggingface.co/datasets/openai/gsm8k https://huggingface.co/datasets/inaciose/smol-smoltalk-pt-v1

Training

Supervised fine-tuning was performed on the Modal cloud platform using:

• 4 × NVIDIA A100 SXM4 80GB

The model was initialized directly from the pretrained checkpoint of inacio-d16v1.

Intended Uses

This model is intended for:

• conversational assistants
• experimentation
• educational purposes
• research on European Portuguese language models

Current Capabilities

The model is capable of:

• maintaining multi-turn conversations
• following simple instructions
• generating structured answers
• responding primarily in Portuguese

Compared to the base model, it produces considerably more natural conversational responses and follows instruction formats learned during supervised fine-tuning.

Current Limitations

This is the first public SFT release of nanochatpt.

The model still presents significant limitations, including:

• factual hallucinations
• weak mathematical reasoning
• inconsistent scientific explanations
• occasional repetition
• occasional generation of special conversation tokens (e.g. <|assistant_end|>)

These behaviours are expected for a first-generation instruction-tuned model trained on a relatively small Portuguese conversational corpus.

Future releases may improve both the instruction dataset and the amount of Portuguese pretraining data.

Requeriments

nanochatpt or nanochat cloned to your machine, install requirements. https://github.com/inaciose/nanochatpt

copy the model to your local folder copy the tokenizer to your local folder

cache/modal ├── chatsft_checkpoints │   └── d16 │   ├── meta_0XXXXX.json │   ├── model_0XXXXX.pt │   ├── optim_0XXXXX_rank0.pt │   ├── optim_0XXXXX_rank1.pt │   ├── optim_0XXXXX_rank2.pt │   └── optim_0XXXXX_rank3.pt └── tokenizer ├── token_bytes.pt └── tokenizer.pkl

export NANOCHAT_BASE_DIR=cache/modal

run the python -m scripts/chat_cli.py

Project

nanochatpt is an personal open research project whose objective is to train compact language models from scratch with European Portuguese as the primary language.

Future work includes:

• larger pretraining corpora
• improved Portuguese instruction datasets
• Portuguese evaluation benchmarks
• additional supervised fine-tuning
• reinforcement learning alignment