indomla-1024

IndoMLA-1024

IndoMLA-1024 is an experimental Indonesian decoder-only causal language model trained from scratch using an MLA-inspired low-rank key/value attention mechanism.

The model contains approximately 139.7 million parameters and supports a maximum context length of 1,024 tokens.

This project is intended primarily for research into Indonesian language modeling, efficient attention architectures, and controlled comparison between conventional Multi-Head Attention and low-rank KV attention designs.

Important: IndoMLA-1024 is a base pretrained language model. It is not instruction-tuned, chat-tuned, or RLHF-aligned.

---

Model Summary

---

Architecture

IndoMLA-1024 follows a decoder-only Transformer architecture.

Each Transformer block consists of:

Input
  │
  ├── RMSNorm
  │
  ├── MLA-inspired Causal Self-Attention
  │
  ├── Residual Connection
  │
  ├── RMSNorm
  │
  ├── SwiGLU Feed-Forward Network
  │
  └── Residual Connection

The full network contains 18 Transformer blocks.

A final RMSNorm is applied before the language-modeling head.

---

MLA-Inspired Attention

The model uses an MLA-inspired attention mechanism.

The term MLA-inspired is used intentionally because this implementation explores the idea of compressing key/value representations into a shared low-dimensional latent space, rather than attempting to reproduce every component of architectures such as DeepSeek MLA.

The query projection remains full-rank:

Hidden State
     │
     ▼
Linear
768 → 768
     │
     ▼
     Q

Keys and values are generated through a shared compressed latent representation:

Hidden State
     │
     ▼
KV Compression
768 → 256
     │
     ▼
KV Latent
256 dimensions
     │
     ├───────────────┐
     │               │
     ▼               ▼
K Projection      V Projection
256 → 768         256 → 768
     │               │
     ▼               ▼
     K               V

The latent rank is:

256

compared with the model hidden size of:

768

This creates a low-rank bottleneck for key/value representations.

---

Attention Dimensions

The attention configuration is:

Hidden size     = 768
Attention heads = 12
Head dimension  = 64

because:

768 / 12 = 64

The projected Q, K, and V representations are reshaped into 12 attention heads before causal scaled dot-product attention is performed.

---

Rotary Position Embeddings

IndoMLA-1024 uses Rotary Position Embeddings (RoPE).

Configuration:

RoPE theta = 10000

RoPE is applied to the query and key vectors before attention score computation.

The model does not rely on a learned absolute positional embedding table.

The configured maximum sequence length is:

1024 tokens

---

RMSNorm

The model uses RMSNorm rather than standard LayerNorm.

The Transformer follows a pre-normalization structure conceptually equivalent to:

x = x + attention(rms_norm(x))
x = x + mlp(rms_norm(x))

A final RMSNorm is also applied after the Transformer stack.

---

SwiGLU Feed-Forward Network

Each Transformer block uses a SwiGLU feed-forward network.

The dimensions are:

768 → 2304 → 768

Conceptually:

Input
  │
  ├── Linear → SiLU
  │
  └── Linear
        │
        ▼
 Element-wise multiplication
        │
        ▼
 Output projection
        │
        ▼
      Output

A simplified mathematical representation is:

SwiGLU(x) = W2(SiLU(W1(x)) ⊙ W3(x))

---

Weight Tying

IndoMLA-1024 ties the token embedding matrix with the output language-modeling head.

Conceptually:

lm_head.weight = token_embedding.weight

This allows the model to reuse the same parameter matrix for input token representations and output token logits.

---

Parameter Count

The model contains:

139,718,400 parameters

or approximately:

139.7M parameters

This places IndoMLA-1024 in the small language model research regime.

The goal of the project is not to compete directly with large production-scale language models, but to provide a manageable architecture for studying Indonesian language modeling and alternative attention designs.

---

Tokenizer

The model uses a custom tokenizer with:

The tokenizer is loaded from:

tokenizer.json

Documents are tokenized and separated using an EOS token before being packed into fixed-length token sequences.

For causal language modeling, training samples conceptually contain:

input:

token_1 token_2 token_3 ... token_1024

target:

token_2 token_3 token_4 ... token_1025

The model therefore learns to predict the next token from the preceding context.

---

Training Dataset

IndoMLA-1024 uses the Indonesian configuration of:

uonlp/CulturaX

with language configuration:

id

The dataset is accessed through Hugging Face dataset streaming.

Streaming allows the training pipeline to process large web-scale datasets without loading the full dataset into system memory or local storage.

---

Dataset Filtering

Before tokenization and training, CulturaX samples pass through a custom filtering pipeline.

The notebook implementation includes filtering intended to remove or reduce low-quality web content.

Filtering criteria include checks related to:

• text length,
• valid text content,
• domain quality,
• URLs,
• undesirable domains,
• low-quality keywords,
• repetitive text,
• excessive symbols,
• malformed text,
• lexical diversity,
• spam-like content,
• SEO-style content,
• commercial boilerplate,
• URL/text consistency,
• word count,
• and manually defined blacklist patterns.

The purpose of this filtering stage is to improve the effective quality of the Indonesian pretraining corpus.

Filtering cannot guarantee that all low-quality, harmful, duplicated, biased, or factually incorrect content has been removed.

---

Training Objective

The model is trained using standard autoregressive causal language modeling.

The objective is:

P(x1, x2, ..., xn)

=

∏ P(xt | x1, ..., xt-1)

At each position, the model predicts the next token conditioned on all preceding tokens.

Training loss is computed using token-level cross entropy.

---

Training Configuration

The main training configuration is:

The effective number of tokens processed per optimizer update is:

micro_batch_size
× gradient_accumulation
× context_length

16 × 2 × 1024

= 32,768 tokens

---

Training Target

The notebook configures a main training target of:

5,000 optimizer steps

If all 5,000 optimizer steps are completed, the nominal training token budget would be:

5,000 × 32,768

which is:

163,840,000 tokens

or approximately:

163.8M training tokens

This number represents the nominal number of token positions processed by the optimizer configuration and should not necessarily be interpreted as the number of unique tokens in the underlying dataset.

---

Learning Rate Schedule

The training process uses:

Warmup
   ↓
Peak LR
   ↓
Cosine Decay
   ↓
Minimum LR

Configured values:

Peak learning rate = 3e-4

Minimum learning rate = 3e-5

Warmup steps = 20

The optimizer is AdamW with:

beta1 = 0.9
beta2 = 0.95
weight_decay = 0.1

Gradient norm clipping is applied with:

max_grad_norm = 1.0

---

Gradient Checkpointing

Gradient checkpointing is enabled during training.

Instead of storing every intermediate activation required for backpropagation, selected activations are recomputed during the backward pass.

Conceptually:

Without checkpointing:

More activation memory
Less recomputation


With checkpointing:

Less activation memory
More recomputation

This allows longer sequences or larger batches to fit within available GPU memory.

---

Training Hardware

The training notebook records an AMD GPU environment using ROCm/HIP.

The environment includes approximately:

GPU VRAM: ~48 GB

The recorded software environment includes:

Python 3.10
PyTorch 2.9 development build
ROCm / HIP 7.2

Training precision:

bfloat16

PyTorch exposes ROCm GPUs through its CUDA-compatible interface, so code may still use APIs such as:

torch.cuda

even when the physical GPU is AMD.

---

Training Progress

Before meaningful training, the initial model loss was approximately:

9.83

This is expected for a randomly initialized language model with a vocabulary of this scale.

The initial smoke-training run showed decreasing loss:

The longer training run continued reducing training loss.

Representative values recorded during training include:

These values represent training loss, not standardized downstream evaluation results.

Loss fluctuations between individual logging intervals are normal during stochastic optimization.

---

Evaluation Status

The current notebook primarily records optimization and training-loss behavior.

No standardized downstream benchmark suite is reported in this model card.

Therefore, the current model card does not claim performance on tasks such as:

• reasoning,
• question answering,
• summarization,
• instruction following,
• translation,
• coding,
• factual recall,
• or standardized Indonesian NLP benchmarks.

A controlled evaluation protocol should be performed before making capability claims.

---

Checkpointing

The training pipeline saves checkpoints periodically.

The configured checkpoint interval is:

500 optimizer steps

Checkpoint data includes information such as:

{
    "run_id": ...,
    "global_step": ...,
    "model_state_dict": ...,
    "optimizer_state_dict": ...,
    "config": ...,
    "train_loss": ...,
    "val_loss": ...,
    "block_size": ...,
    "micro_batch_size": ...,
    "grad_accum_steps": ...,
    "tokens_per_step": ...,
    "created_at": ...
}

Saving model configuration alongside model weights makes experiments easier to reproduce and inspect.

---

Loading a Native Checkpoint

IndoMLA-1024 currently uses a custom PyTorch architecture.

The checkpoint should therefore be restored using the same model implementation used during training.

Example:

import torch

checkpoint = torch.load(
    "checkpoint.pt",
    map_location="cpu"
)

config = IndoMLAConfig(**checkpoint["config"])

model = IndoMLAForCausalLM(config)

model.load_state_dict(
    checkpoint["model_state_dict"]
)

model.eval()

print("Checkpoint step:", checkpoint["global_step"])

The custom classes:

IndoMLAConfig
IndoMLAForCausalLM

must be available before loading the checkpoint.

---

Loading the Tokenizer

The tokenizer can be loaded using the Hugging Face tokenizers library.

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file(
    "tokenizer.json"
)

text = "Indonesia adalah negara"

encoding = tokenizer.encode(text)

print(encoding.ids)

---

Generation

Because IndoMLA-1024 is a causal language model, generation proceeds autoregressively:

Prompt
  ↓
Predict next token
  ↓
Append token
  ↓
Predict next token
  ↓
Append token
  ↓
...

The current implementation uses a custom PyTorch model, so inference should use the architecture defined in the accompanying notebook or source code.

The model should not automatically be assumed to work with:

AutoModelForCausalLM.from_pretrained(...)

unless the repository is subsequently packaged with the necessary Hugging Face Transformers-compatible configuration and modeling files.

---

Intended Use

IndoMLA-1024 is intended primarily for research and experimentation.

Appropriate use cases include:

• Indonesian language modeling research,
• decoder-only Transformer experiments,
• MLA-inspired architecture experiments,
• low-rank key/value representation research,
• comparison with standard Multi-Head Attention,
• pretraining experiments,
• tokenizer experiments,
• continued pretraining,
• supervised fine-tuning research,
• representation learning,
• architecture ablation studies,
• educational experimentation,
• and reproducibility research.

---

Not Intended As a Chat Model

IndoMLA-1024 is a base pretrained model.

The model has not been trained specifically to follow instructions such as:

Explain this concept.

Summarize this article.

Answer this question.

Write Python code.

Base-model completion behavior is more appropriate.

For example:

Indonesia adalah negara

may be continued autoregressively based on patterns learned during pretraining.

Instruction-following behavior would require additional training such as:

Base Pretraining
      ↓
Supervised Fine-Tuning
      ↓
Instruction Model

Potential alignment stages could additionally include preference optimization or other alignment methods.

---

Limitations

IndoMLA-1024 is an experimental research model and has several important limitations.

Small Model Scale

At approximately:

139.7M parameters

the model is substantially smaller than modern production-scale large language models.

Its capabilities should therefore not be compared directly with multi-billion-parameter instruction-tuned systems without accounting for model scale, training compute, dataset size, and post-training.

---

Limited Training Budget

The experiment represents relatively early-stage pretraining.

The configured 5,000-step run corresponds to approximately:

163.8M processed token positions

which is not necessarily a compute-optimal training budget for a model of this size.

Additional training may significantly change model behavior.

---

No Instruction Tuning

The model has not undergone supervised instruction tuning.

Therefore, it should not be expected to reliably:

• follow commands,
• answer questions conversationally,
• reject harmful prompts,
• maintain dialogue state,
• or produce assistant-like responses.

---

No Alignment Training

The model has not undergone alignment procedures such as:

• RLHF,
• DPO,
• preference optimization,
• safety tuning,
• or human-feedback alignment.

Generated content may therefore be inappropriate, incorrect, biased, unsafe, or otherwise undesirable.

---

Web-Derived Training Data

CulturaX is derived from large-scale web data.

Although a custom filtering pipeline is applied, web-derived datasets can contain:

• factual inaccuracies,
• outdated information,
• social biases,
• offensive content,
• duplicated text,
• misinformation,
• spam,
• and low-quality language.

Generated text should not automatically be considered factual.

---

No Established Benchmark Results

The current experiment does not provide comprehensive standardized downstream evaluation.

Training loss alone is insufficient for determining:

• reasoning capability,
• factual knowledge,
• language understanding,
• generation quality,
• or downstream task performance.

---

Research Motivation

The main research motivation behind IndoMLA is to study whether compressed key/value representations can provide an interesting alternative to conventional Multi-Head Attention for relatively small Indonesian language models.

A simplified comparison is:

Conventional Multi-Head Attention

Hidden State
   │
   ├──────► Q
   │
   ├──────► K
   │
   └──────► V

versus:

IndoMLA-1024

Hidden State
   │
   ├────────────────────► Q
   │
   ▼
Compressed KV Latent
   │
   ├────────────────────► K
   │
   └────────────────────► V

The KV latent dimension is:

256

while the hidden dimension is:

768

This architecture creates an explicit low-rank bottleneck for the key/value pathway.

---

Why "MLA-Inspired"?

This repository deliberately uses the terminology:

MLA-inspired

rather than claiming a complete implementation of Multi-head Latent Attention as used in larger architectures.

The central idea implemented here is the shared low-rank compression of key/value representations.

The query path remains full-rank.

Therefore, a more precise description of the architecture is:

A decoder-only Transformer with full-rank query projections and shared low-rank latent key/value projections.

This terminology helps distinguish the experimental design from complete implementations of other MLA architectures.

---

Comparison Research

The architecture is suitable for controlled comparison with a conventional MHA baseline.

A rigorous comparison should match important experimental variables, including:

Tokenizer
Dataset
Data ordering
Context length
Parameter budget
Training tokens
Optimizer
Learning rate
Batch size
Precision
Evaluation protocol

Possible measurements include:

• validation loss,
• perplexity,
• training throughput,
• tokens per second,
• peak VRAM consumption,
• inference latency,
• KV-cache memory usage,
• parameter efficiency,
• and downstream benchmark performance.

Without matched experimental conditions, architectural conclusions should be treated cautiously.

---

Reproducibility

The reference notebook documents the experiment end-to-end, including:

• environment configuration,
• dataset loading,
• CulturaX streaming,
• custom text filtering,
• tokenizer loading,
• token packing,
• model architecture,
• MLA-inspired attention,
• RoPE,
• RMSNorm,
• SwiGLU,
• parameter counting,
• optimizer setup,
• learning-rate scheduling,
• gradient accumulation,
• gradient checkpointing,
• checkpoint saving,
• and training progress.

Reference notebook:

indomla-1024.ipynb

Repository:

https://github.com/Rai7/indomla-1024

---

Experimental Status

Current status:

Experimental Research Base Model

The project should currently be interpreted as an architecture and pretraining experiment rather than a production-ready language model.

Future work may include:

• longer pretraining,
• validation-loss evaluation,
• MHA baseline comparison,
• controlled architecture ablations,
• inference throughput measurement,
• memory-efficiency analysis,
• Indonesian benchmark evaluation,
• supervised fine-tuning,
• and Hugging Face Transformers integration.

---

Citation

If you use this model or implementation in research, please cite the repository.

A formal citation entry can be added once the associated research paper or technical report receives its final publication metadata.

---

License

IndoMLA-1024 is released under the Apache License 2.0.

See the repository license for additional details.

---

Disclaimer

This model is provided for research and educational purposes.

Model outputs may contain incorrect, biased, offensive, misleading, or unsafe information.

Users are responsible for evaluating outputs and determining whether the model is appropriate for their intended application.