Alo-70m-Base

Alo-70M-Base

Model Summary

Alo-70M-Base is a compact, 69-million parameter foundational Large Language Model (LLM) built exclusively for the Bengali (Bangla) language. Trained entirely from scratch, it aims to lower the compute barrier for Bengali NLP research and provide a viable path for deploying localized AI assistants on standard CPUs and resource-constrained edge devices.

Despite its ultra-lightweight footprint, Alo-70M-Base achieves competitive parity on localized reasoning tasks against significantly larger cross-lingual models (such as 270M and 1B parameter baselines).

• Developer: Fahad Hossain
• Language: Bengali (Bangla)
• Model Type: Causal Language Model (Autoregressive Transformer)
• Parameter Count: 69 Million
• License: Apache 2.0

Related Resources

This model is the base foundational model. We also release the instruction-tuned version, the synthetic dataset used for alignment, and the standalone tokenizer:

• Instruction-Tuned Model: spitfire4794/Alo-70M
• Alignment Dataset: spitfire4794/Bangla-SFT-50k
• Tokenizer: spitfire4794/beng_bpe

Usage

You can load and generate text with Alo-70M-Base using the transformers library.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70M-Base"
tokenizer_id = "spitfire4794/beng_bpe"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(tokenizer_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

prompt = "বাংলাদেশের রাজধানী ঢাকা"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=50, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Details

Alo-70M-Base is based on a scaled-down LLaMA transformer architecture optimized for hardware alignment and parameter efficiency.

• Layers: 12
• Hidden Dimension ($d_{model}$): 512
• Intermediate FFN Dimension: 1408
• Attention Mechanism: Grouped-Query Attention (GQA) with 8 query heads and 4 key-value heads.
• Positional Embeddings: Rotary Position Embeddings (RoPE) with a base frequency theta of 10,000.
• Activation Function: SwiGLU (silu)
• Word Embeddings: Untied (tie_word_embeddings = False). The static lookup tables account for 48.6% (33.5M) of the total parameter budget.
• Context Window: 1024 tokens.
• Tokenizer: Custom Bengali Byte-Pair Encoding (BPE) tokenizer (spitfire4794/beng_bpe) with a highly optimized vocabulary size of 32,768 (2^15) designed to maximize TPU efficiency.

Pre-Training Data

The model was pre-trained on a curated, strictly Bengali corpus containing approximately 19.25 billion tokens. The dataset is a blend of:

1. FineWeb-2 (Bengali Split): Providing diverse, web-scale filtered documents.
2. TituLM Bengali Corpus (CommonCrawl Split): Introducing balanced, general-domain web data.

Training Procedure

Alo-70M-Base was initialized with completely randomized weights (no transfer learning) and trained from scratch.

• Hardware: Kaggle Tensor Processing Unit (TPU) v5e-8
• Training Time: 12.9 hours (wall time) across two sequential runs.
• Optimizer: AdamW (adam_torch_xla) with beta_1 = 0.9, beta_2 = 0.95, epsilon = 10^-8
• Weight Decay: 0.1
• Learning Rate Schedule: Cosine Decay with a gradual warmup phase, peaking at 6 times 10^-4.
• Precision & Optimization: Mixed bfloat16 (bf16) precision with gradient checkpointing enabled to maximize throughput.

Evaluation Results

The model was evaluated in a zero-shot setting across various Bengali reasoning and knowledge benchmarks using a continuation-based evaluation methodology (calculating conditional log-probabilities directly over raw Bengali text to avoid "pointer-reasoning" bottlenecks).

Note: Alo-70M-Base matches or outperforms both the 270M and 1B baselines on CommonsenseQA and PIQA despite having 4x to 14x fewer parameters.

Limitations and Biases

• Encyclopedic Knowledge: Due to the severe parameter constraints (<100M parameters), the model physically lacks the parametric capacity to act as a deep encyclopedic knowledge base, which reflects in its performance on knowledge-intensive retrieval tasks like MMLU and ARC-Challenge.
• Tokenizer Evaluation Bias: During continuation-based log-likelihood evaluations, Alo-70M's highly efficient native tokenizer may possess a mathematical advantage over multilingual baselines (which heavily fragment Bengali text), contributing partially to its high performance margins on specific tasks.
• Instruction Following: This is a base model. It will attempt to complete text rather than follow conversational instructions. For chat and instruction-following, please use Alo-70M.

Citation

Technical paper out soon.