smolagent-ms-800m-base

SmolAgent-MS 800M Base

Dense decoder-only Transformer trained from scratch on 8.0B tokens of Malay-domain text (97% Malay / 43 domains incl. Hansard, laws, news, wiki).

Architecture

• 24 layers, hidden 2048, GQA (16Q/4KV, head 128), SwiGLU (2816), RMSNorm, RoPE, tied embeddings, vocab 65,536 (custom byte-level BPE), context 8192
• ~801M parameters total

Training

• WSD schedule: 2% warmup, peak LR 3e-4 (sweep-selected), 15% linear decay
• 61,035 steps x 131,072 tokens (bf16 + cut-cross-entropy, ~40k tok/s on H200)
• Decontaminated: 13-gram document-level filter against all held-out eval sets
• 67 loss-spike events auto-skipped/rolled-back; log in spike_log.jsonl (intended as curation signal for synthetic pretraining)

Results (likelihood-scored, base model)

English benchmarks are near chance by design: this run's mix was Malay-only (the English corpus finished processing after launch). The continued-pretraining phase adds English + structured data.

Usage

Base model - not instruction tuned. Expect continuation-style generation, multiple Malay registers (encyclopedia/news/colloquial/procedural).

Custom architecture: load with the project repo khursanirevo/smolagent-ms (src/model/).