LFM2.5-50M-Vietnamese

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>

LFM2.5 50M Vietnamese — Continued Pre-Training

A 48M-parameter model based on LiquidAI/LFM2.5-1.2B-Instruct, scaled down and further pre-trained on a large Vietnamese text corpus. LFM2.5 is a hybrid conv-attention architecture from Liquid AI, designed for efficient inference with linear-time complexity in sequence length.

Model Details

Training

Continued Pre-Training

The base architecture from LiquidAI/LFM2.5-1.2B-Instruct was scaled down to 50M parameters (9 layers, hidden_size=256) and then continued pre-training was performed on Vietnamese text.

• Base model: LiquidAI/LFM2.5-1.2B-Instruct (random initiated 50M params weight)
• Dataset: Tuan-NT/vietnamese-corpus-pretrain -- 11.3M samples of Vietnamese text (news, Wikipedia, web crawl)
• Tokens seen: 2.46 billion
• Steps: 178,000 (plateau reached around 120K steps)
• Batch size: 8 x 4 gradient accumulation = effective 32
• Sequence length: 512 tokens
• Learning rate: 2e-4 (cosine schedule)
• Optimizer: AdamW
• Hardware: NVIDIA RTX 5080 16GB
• Training time: ~21 hours
• Framework: Unsloth + HuggingFace Transformers

Loss Curve

Loss plateaued around step 120K at ~4.08. The model was trained for 0.5 epochs of the full dataset (2.46B tokens out of ~5.8B total).

Intended Use

This model is intended as a backbone for downstream Vietnamese NLP tasks, not as a standalone text generator. At 48M parameters it is too small for general-purpose generation but well-suited as a feature extractor or for fine-tuning on specific tasks.

Recommended applications:

• Named Entity Recognition (NER) -- add a token classification head and fine-tune on labeled data
• Text classification -- add a classification head for sentiment, intent, or category prediction
• Query understanding -- parse e-commerce search queries into structured components
• Lightweight inference -- the hybrid conv-attention architecture enables fast CPU inference (~2ms per query at 20 tokens)
• On-device NLP -- small enough for mobile/edge deployment

Not recommended for:

• Open-ended text generation (too small for coherent generation)
• Tasks requiring world knowledge (limited capacity)
• Multilingual tasks (Vietnamese-focused pre-training)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "GazTrab/lfm2.5-50m-vietnamese",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("GazTrab/lfm2.5-50m-vietnamese")

# Get hidden states for downstream tasks
inputs = tokenizer("dien thoai samsung galaxy", return_tensors="pt")
outputs = model(**inputs, output_hidden_states=True)
last_hidden = outputs.hidden_states[-1]  # [1, seq_len, 256]

# Or use as a token classifier backbone
# See: https://huggingface.co/docs/transformers/tasks/token_classification

Background

This model was developed as part of a search system project. The goal was to build a lightweight NER backbone that could:

1. Parse Vietnamese search queries with typos, missing diacritics, slang, and abbreviations
2. Run inference in under 2ms on CPU for real-time search
3. Handle different entity classification

The LFM2.5 architecture was chosen after benchmarking against Qwen2.5 and Qwen3.5 at similar parameter counts, as it achieved the best speed-accuracy tradeoff with its hybrid conv-attention design.

Citation

@misc{LFM2.5-50M-Vietnamese,
  title={LFM2.5 50M Vietnamese: Continued Pre-Training of LFM2.5 on Vietnamese Corpus},
  author={GazTrab},
  year={2026},
  url={https://huggingface.co/GazTrab/LFM2.5-50M-Vietnamese}
}

Acknowledgments

• Liquid AI for the LFM2.5 architecture (LiquidAI/LFM2.5-1.2B-Instruct)
• Tuan-NT for the Vietnamese pre-training corpus
• Unsloth for efficient training framework