synthpre-557m

synthpre-557m

Base language model for Bahasa Melayu (Malaysian Malay). Dense Llama-style transformer, 557.3M parameters, RoPE/SwiGLU/RMSNorm, tied embeddings, 1024-token context, Qwen2.5 tokenizer (151,936 vocab).

⚠️ Early checkpoint — undertrained

This is a mid-training checkpoint of an in-progress run, published for internal use:

• trained ~2.1B of a planned 14.2B tokens (~15% of schedule)
• val loss 2.37 (PPL ~10.7) and still falling at export time
• the full run targets val loss ~2.2-2.4; this export will be superseded

Training data

All training data was collected and curated by us — an in-house Malaysian-web crawl and manual curation pipeline, plus hand-reviewed government/hansard/legal documents. We do not use any third-party text collections. 13.88B tokens, 10.85M documents. Pipeline: manual review calibration (182 hand-labeled documents for language identification), Indonesian contamination removal (~20% of raw), quality filtering, exact + MinHash deduplication (within and across pulls), domain-disjoint train/eval/heldout splits. No synthetic data (measured neutral at scale — see repo report).

Training setup

• 2× H200 DDP, bf16 autocast, ~86k tok/s
• AdamW (0.9, 0.95, wd 0.1), LR 4e-4 (measured via 4-rate sweep + 600-step validation), WSD schedule (2% warmup / stable / 15% decay), grad clip 1.0
• batch 524,288 tokens/step

Intended use

Base model for: Malay language understanding, continued pretraining, instruction/tool-calling fine-tuning (BFCL-JSON format), Malaysian-domain applications. Not instruction-tuned; expect base-model behavior (completion, not conversation). English capability is incidental only — the training mix is effectively 100% Malay.

Limitations

• Undertrained (see above)
• 1k context only (8k midtraining phase planned)
• No code, minimal English, no safety alignment
• Knowledge limited to corpus content; facts learned only where attested

Evaluation

Val loss 2.3658 on heldout Malay web text (domain-disjoint). Zero-shot bfcl-ms (Malay function-calling) baseline pending; instruct/tool SFT phases planned on top of this base.