indomla-512

IndoMLA-512

An experimental Indonesian decoder-only language model trained from scratch on a single consumer GPU, including its tokenizer.

This repository holds the MLA-inspired checkpoint from the research project IndoChat-Scratch: A Consumer-GPU Study of MHA and MLA-Inspired 150M-Parameter Decoder-Only Language Models for Indonesian. Its control is the standard-attention baseline at RaiRamones/indomha-512.

⚠️ Early-stage checkpoint — 30,000 of 178,000 planned steps (~16.9%), ~1.97B tokens processed. This is a base model: no instruction tuning, no RLHF, no safety alignment. It predicts the next token; it does not follow instructions.

Model Details

Implemented manually in PyTorch — not an AutoModelForCausalLM architecture, and not initialized from any pretrained model.

Attention Architecture

Rather than storing full per-head keys and values, the model compresses hidden states into a shared latent representation at rank r_kv = 448 (= d_model / 2), then projects that latent into per-head content keys and values. Queries pass through a matching low-rank bottleneck at r_q = 448.

Each 64-dim head splits into 32 content (NoPE) dimensions and 32 RoPE dimensions, with the rotary key component generated independently and shared across all 14 heads. This keeps the position-dependent component separate from the position-independent compressed KV latent.

The KV cache therefore stores 448 latent values + 32 shared rotary-key values = 480 values per token per layer, versus 1,792 for standard MHA.

Why "MLA-inspired"? The implementation preserves the latent-cached decoding and its memory advantage, but does not implement DeepSeek-style absorption of the latent up-projections into fused attention matmuls — per-head representations are reconstructed at each decode step. Do not read this as an exact reproduction of a production MLA implementation.

Training

Trained on a single RTX 4060 Laptop GPU (8 GB VRAM) with bf16 mixed precision.

Corpus: Indonesian Wikipedia (wikimedia/wikipedia, 20231101.id) + CulturaX — 11,648,042,123 train / 118,020,446 validation tokens. Preprocessing covers NFKC normalization, HTML and URL stripping, repeated-character normalization, Indonesian-language heuristic filtering, minimum-length filtering, and exact deduplication via 64-bit BLAKE2b hashing. Validation split drawn probabilistically at p=0.01.

The tokenizer reserves eight special tokens — <pad>, <unk>, <bos>, <eos>, plus four chat markers held for possible future SFT. Documents are encoded as <bos> … <eos> before concatenation into the token stream.

The MHA and MLA runs share the same corpus, tokenizer, and optimization settings.

Known corpus issue: this checkpoint's training data contains some Indonesian online-gambling and web-spam content. A stronger filter was added after the corpus was built, so generations may occasionally reproduce those patterns. Both checkpoints in the comparison are affected equally.

Evaluation

Corrected full-validation evaluation over 5,000 non-overlapping windows (~2.56M tokens), sampled across the complete validation split with identical samples for both architectures.

The original in-training estimator reported loss 1.5669 / PPL 4.792 — do not use those numbers. It sampled overlapping windows clustered near the start of the validation array, covering only a small and unusually easy region of the corpus.

Finding: the 0.39% perplexity gap is small, and checkpoint rankings flipped repeatedly during training — so this experiment does not establish a language-modeling quality advantage for either architecture. The meaningful difference is the trade-off: MLA needs ~3.73× less KV cache, which under a fixed 6 GB KV budget is roughly 871 vs. 233 concurrent sequence caches. That memory advantage costs ~30% decode throughput, since the unoptimized implementation reconstructs per-head representations from latent states without projection absorption or fused kernels.

Qualitative scores

Manual scoring on a small Indonesian prompt suite covering continuation, factual completion, instruction following, arithmetic, paragraph generation, ML explanation, list generation, and story continuation:

Small hand-scored suite, not a standardized benchmark.

Intended Use

Research on Indonesian language modeling, decoder-only architectures, MLA-inspired attention and MHA/MLA ablations, KV-cache efficiency, tokenizer design, and consumer-GPU training. Also usable as a compact Indonesian base model for experimental continuation training.

Out of scope: medical, legal, or financial advice; credit decisions; autonomous or safety-critical systems; any factual information system without verification.

Limitations

• Early checkpoint (16.9% of schedule) — not final architecture performance.
• Single seed (1337) — small MHA/MLA differences may be seed variance.
• No standardized Indonesian downstream benchmark yet — no IndoNLU or IndoLEM results; evaluation is perplexity, throughput, memory, and the small qualitative suite above.
• Corpus contamination — see the note above.
• Hallucination — output may be fabricated, inconsistent, or unsupported by the corpus.
• Repetition and drift in longer generations, expected at this scale and stage.
• Weak instruction following — apparent compliance is an artifact of instruction-like text in the corpus, not a capability.
• Limited reasoning — arithmetic, multi-step logic, planning, and factual QA are unreliable.
• Implementation-dependent benchmarks — one PyTorch implementation, one GPU, one environment. Not universal characteristics of MLA.

Reproducibility

seed 1337 · context 512 · vocab 16,000 · AdamW
LR 2e-4 → 2e-5 · warmup 2,000 · cosine decay
micro batch 2 · grad accum 64 · 65,536 tokens/step
bfloat16 · RTX 4060 Laptop 8GB

The broader project holds the training configs for both architectures, the manual PyTorch implementations, the tokenizer and its training scripts, the data-cleaning and tokenization pipeline, training metrics, step-aligned comparisons, the corrected evaluation, the KV-cache implementation with correctness tests, and inference benchmarks. Treat this as a research checkpoint inside a controlled experiment, not a standalone production model.

Future Work

Continue to 60K, 90K, 120K, and the full 178K-step schedule; additional random seeds; standardized Indonesian downstream evaluation; improved validation protocols; MLA projection absorption and fused attention kernels; longer context; higher-concurrency inference benchmarks; quantized inference; Indonesian SFT and instruction-tuned variants; larger models on cloud GPUs.

Citation

@misc{muslim2026indochatscratch,
  title        = {IndoChat-Scratch: A Consumer-GPU Study of MHA and MLA-Inspired 150M-Parameter Decoder-Only Language Models for Indonesian},
  author       = {Muslim, Azis},
  year         = {2026},
  note         = {Early-stage pretraining study; results at matched 30,000 training steps}
}

A DOI or arXiv identifier can replace this once the paper is archived.

Author & License

Azis Muslim, Independent Researcher — RaiRamones

Apache-2.0. Users are responsible for ensuring downstream use complies with applicable laws, regulations, and dataset licenses.

Disclaimer: experimental research model. May generate incorrect, biased, repetitive, or fabricated content. Verify outputs before any real-world use.