talos-mini-1m-oasst1

Talos Mini 1M — OASST1

Experimental 1,000,320-parameter Talos Mini language model trained on a small subset of OpenAssistant/oasst1.

This repository contains the native Talos checkpoint and native Talos Byte-Level BPE tokenizer.

Model specifications

• Parameters: 1,000,320
• Architecture: TalosGPT decoder-only transformer
• Hidden size: 128
• Layers: 3
• Attention heads: 8
• KV heads: 4
• Head dimension: 16
• Dense FFN intermediate size: 512
• Vocabulary size: 1,024
• Maximum sequence length: 512
• Tokenizer merges: 764
• Training data: 1,800 OASST1 messages
• Evaluation data: 200 held-out OASST1 messages
• Training sequence length: 64
• Training batch size: 4
• Optimizer: AdamW
• Learning rate: 3e-3

Training

The model was trained using the existing Talos training stack.

Published checkpoint:

step-26925.pt

The checkpoint contains the native Talos model state and training metadata.

Results

At step 26,925:

• Validation loss: approximately 2.254
• Validation perplexity: approximately 9.52
• Validation next-token accuracy: approximately 35%
• Evaluated tokens: 102,816

These results come from a small research experiment and are not a benchmark of general language-model capability.

The model can generate recognizable language patterns, but it is not a coherent general-purpose conversational model.

Tokenizer

The tokenizer is the native Talos ByteLevelBPETokenizer.

It is not a Hugging Face GPT-2 tokenizer conversion.

The tokenizer uses Talos-native byte IDs and merge IDs.

• Vocabulary: 1,024
• Merges: 764

Status

Demonstrated

• 1M-parameter Talos model construction
• Native Talos tokenizer
• OASST1 data loading
• Train/evaluation split
• Training
• Checkpoint saving/loading
• Held-out evaluation
• Text generation
• KV-cache equivalence testing
• Greedy generation

Limitations

This is an experimental dev-scale model trained on a small dataset and short sequence length.

It is not intended to compete with modern general-purpose language models.

The 1M model is a scaling experiment between the 254K Talos Mini prototype and larger future Talos configurations.

Long-term Talos scaling targets are research goals and are not represented here as demonstrated capabilities.