Alo-70m

Alo-70M (Instruct)

Model Summary

Alo-70M is the instruction-tuned version of the ultra-lightweight 69-million parameter Bengali language model, Alo-70M-Base. Built on a scaled-down LLaMA architecture, it is designed to act as a highly efficient, edge-deployable localized AI assistant.

Fine-tuned on a curated dataset of instruction-response pairs using the ChatML format, Alo-70M is aligned for tasks such as summarization, entity extraction, text editing, and question answering in native Bengali. Despite its compact footprint, it offers a viable path for edge AI deployment on standard CPUs and mobile hardware.

• Developer: Fahad Hossain
• Language: Bengali (Bangla)
• Model Type: Causal Language Model (Instruction-Tuned Autoregressive Transformer)
• Parameter Count: 69 Million
• License: Apache 2.0

Related Resources

• Base Model: spitfire4794/Alo-70M-Base
• Alignment Dataset: spitfire4794/Bangla-SFT-50k
• Tokenizer: spitfire4794/beng_bpe

Usage

Alo-70M was trained using the ChatML template. The chat template is built directly into the Jinja template of the tokenizer (spitfire4794/beng_bpe). You can leverage it using Hugging Face's apply_chat_template interface:

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70M"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Define the instruction in ChatML format
messages = [
    {"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
]

# Apply the pre-configured ChatML template
inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=150, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)

# Decode response (omitting user prompt)
response = outputs[0][inputs.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Supervised Fine-Tuning (SFT) Details

Alo-70M was aligned using a curated subset of the Bangla-SFT-50k dataset formatted using ChatML.

• Dataset Pruning: Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample Structured Formatting category was excluded. The final active training mixture consisted of 37,536 aligned pairs.
• Engineering Properties: The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
• Hardware: NVIDIA T4 and L4 GPUs.
• Hyperparameters:
  • Optimizer: Fused AdamW (adamw_torch_fused) with beta_1 = 0.9, beta_2 = 0.999, epsilon = 10^-8
  • Weight Decay: 0.0
  • Learning Rate Schedule: Cosine decay, peaking at 3 times 10^-4 with a 10% linear warmup.
  • Epochs: 3
  • Effective Batch Size: 32 (per-device 8 with gradient accumulation of 4).
  • Precision: Native Automatic Mixed Precision (AMP).

Model Architecture details

Like its base model, Alo-70M utilizes a parameter-efficient architecture:

• Layers: 12
• Hidden Dimension: 512 | Intermediate FFN: 1408
• Attention: Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
• Positional Embeddings: RoPE (Base freq: 10,000)
• Word Embeddings: Untied.
• Context Window: 1024 tokens.

Evaluation Results

The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):

Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.

Limitations and Biases

• Alignment Tax (Catastrophic Forgetting): While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
• Knowledge Retrieval: With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
• Context Length: The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.

Citation

Technical paper out soon.