alpaca_capitals_27m

alpaca_capitals_27m

alpaca_capitals_27m is a small decoder-only language model trained from scratch in Spanish as an educational experiment. It studies how a compact Transformer learns general instruction-following patterns together with a narrow factual domain: country-capital associations.

This release corresponds to the alpaca_capitals experiment and contains approximately 27M parameters. The export script records the exact parameter count detected from the selected checkpoint in export_manifest.json.

Model details

• Language: Spanish
• Architecture: custom decoder-only Transformer
• Approximate size: 27M parameters
• Vocabulary: 10,000-token Byte-Level BPE tokenizer
• Context length: 256 tokens
• Hidden size (d_model): 512
• Transformer layers: 8
• Attention heads: 8
• KV heads: 2
• Training objective: next-token prediction
• Weight format: Safetensors
• Framework: PyTorch / custom training code
• Training hardware: NVIDIA RTX 3070 Laptop GPU
• Released checkpoint: step 18,000 (step_0018000.pt)

NOTE: This repository uses a custom architecture. It is not currently presented as a drop-in transformers.AutoModelForCausalLM.from_pretrained() model. The repository includes the original architecture/training source and the tokenizer/configuration artifacts required to reconstruct the model.

Training data

The model was trained with a mixed Spanish instruction corpus:

1. bertin-project/alpaca-spanish — the general Spanish instruction-following dataset.
2. capitals_clean.jsonl — a local dataset designed to reinforce country-capital knowledge in Spanish.

The capital dataset covers the 195-country/state scope used in this experiment and contains multiple Spanish formulations for each country. During training, the local data were oversampled with local_weight=5.

The upstream bertin-project/alpaca-spanish dataset is published on Hugging Face under CC BY 4.0. Users should review the upstream dataset terms and the terms applicable to any additional local data before redistribution or commercial use.

Training configuration

The selected checkpoint is recorded in export_manifest.json, together with its SHA-256 hash.

Released checkpoint

This repository releases the step 18,000 checkpoint, corresponding to:

step_0018000.pt

The full training run continued to 30,000 steps and produced checkpoints every 2,000 steps. General validation loss and task-specific factual behavior were not perfectly aligned during the experiment.

Among the initially inspected checkpoints, the lowest general validation loss occurred around step 14,000. However, qualitative testing on country-capital questions showed stronger factual behavior from the step 18,000 checkpoint. For that reason, step 18,000 was selected as the model released in this repository.

This checkpoint choice is based on qualitative task-specific testing rather than a complete 195-country benchmark. The exact released checkpoint step and the SHA-256 hash of the exported model.safetensors file are also recorded in export_manifest.json.

Example behavior

Usuario: ¿Cuál es la capital de China?
Modelo: La capital de China es Pekín. La respuesta es Pekín.

Usuario: Dime la capital de Marruecos
Modelo: La capital de Marruecos es Rabat. La respuesta es Rabat.

Usuario: ¿Cuál es la capital de Argentina?
Modelo: La capital de Argentina es Buenos Aires. La respuesta es Buenos Aires.

For factual testing, greedy decoding was more stable than stochastic sampling:

temperature = 0.1
top_k = 1
max_new_tokens = 80

With top_k=1, generation is effectively greedy.

Training curves and checkpoint selection

The following charts summarize the behavior of the full 30K-step training run.

Loss curves

The training loss decreases steadily throughout the run, showing that the model continues fitting the training distribution. The validation-loss checkpoints, however, improve strongly at first and then stop improving at the same pace. This is a useful sign that the model keeps learning patterns from the training data while the general validation objective begins to saturate.

Perplexity

Perplexity drops rapidly during the early stages of training and then stabilizes. Validation perplexity improves substantially at first, reaching its lowest region around the middle part of the run, and then becomes less favorable in later checkpoints. This is consistent with the checkpoint table and indicates that the checkpoint with the lowest global validation loss is not necessarily the best checkpoint for the specific geography task evaluated qualitatively.

Checkpoint milestones

The checkpoint summary shows that the automatic recommendation based on lowest validation loss was:

• step 14,000 with approximately:
  • val_loss = 2.5358
  • val_ppl = 12.63

However, the model released in this repository is:

• step 18,000 (step_0018000.pt)

Why release step 18,000 instead of the automatic recommendation?

The training monitor selected step 14,000 because it had the best general validation loss among the saved checkpoints. That metric is useful, but this project had a narrower target objective as well: answering country-capital questions in Spanish.

During qualitative testing of the released task, the step 18,000 checkpoint showed better practical behavior on country-capital prompts than the automatic 14K recommendation. In particular, it produced stronger answers under the factual decoding configuration used for this experiment:

temperature = 0.1
top_k = 1
max_new_tokens = 80

This means the final release decision was based on task-specific qualitative performance, not only on the generic validation metric shown by the training dashboard. That choice is pedagogically useful because it illustrates an important lesson in machine learning:

the checkpoint with the best global validation loss is not always the checkpoint that performs best on the final task you care about.

In other words, step 14,000 was the best checkpoint according to the monitor’s automatic validation-loss criterion, while step 18,000 was selected for publication because it was more convincing for the intended capital-city inference demonstrations.

Known limitations

This is an educational tiny language model, not a production knowledge system.

Observed limitations include:

• sensitivity to capitalization, accents, punctuation, and phrasing;
• occasional failures when a country name is written in lowercase even when the canonical form succeeds;
• weaker behavior on terse or out-of-distribution prompts;
• hallucination of unrelated facts under some prompt variations;
• no guarantee of correct answers for all countries;
• no exhaustive 195-country accuracy benchmark has yet been reported;
• factual knowledge is stored parametrically and may become outdated;
• special or politically disputed capital-city cases require careful interpretation.

A useful result from this experiment is that learning a canonical factual mapping does not automatically imply robustness to surface-form variation.

Intended use

This model is intended for:

• education about decoder-only Transformers;
• demonstrations of training a language model from scratch;
• experiments on factual memorization in small language models;
• studies of dataset diversity and prompt robustness;
• comparisons between general validation loss and task-specific performance.

It is not intended as an authoritative geography reference or as a production conversational assistant.

Files and Safetensors release

The public model weights are stored as:

model.safetensors

The original experimental checkpoints were PyTorch .pt files. They were converted to Safetensors for safer public distribution. The Safetensors release contains model weights only; optimizer and scheduler states are intentionally excluded.

Typical repository contents:

README.md
model.safetensors
model_config.json
tokenizer.json
tokenizer_stats.json
inference_settings.json
export_manifest.json
checkpoint_index.csv
training_metrics.csv
tiny_spanish_transformer_v2.py
safetensors_loader.py
requirements.txt

Loading the published weights

The weights can be read with:

from safetensors.torch import load_file

state_dict = load_file("model.safetensors", device="cpu")

Because this is a custom Transformer implementation, instantiate the architecture using the included model source/configuration and then load the state dictionary:

model.load_state_dict(state_dict)

See tiny_spanish_transformer_v2.py and model_config.json for the architecture used by the experiment.

Reproducibility

export_manifest.json records the selected source checkpoint, checkpoint step, detected parameter count, SHA-256 of model.safetensors, and serialization format.

Evaluation status

The model has been manually evaluated with several country-capital prompts and prompt variations. A formal benchmark across all 195 countries and multiple surface forms is a recommended next step.

Useful future metrics include canonical capital accuracy, lowercase/no-accent accuracy, terse-prompt accuracy, paraphrase robustness, and accuracy by checkpoint.

Acknowledgements

The general instruction data are based on bertin-project/alpaca-spanish.

This model was developed as a small-scale Spanish Transformer training experiment for educational and research purposes.