mobilellm-pro-fine-tunned

MobileLLM-Pro (P1) Model Card

We are introducing MobileLLM-P1 or Pro, a 1B foundational language model in the MobileLLM series, designed to deliver high-quality, efficient on-device inference across a wide range of general language modeling tasks. <br> We open-source two variants of the model: A pre-trained base model along with quantized checkpoints for CPU and accelerator inference, as well as an instruction tuned version, showing competitive performance against models in the this size range on tasks like tool calling, question answering, rewriting and summarization.

You are currently in the MobileLLM-Pro (instruct) repository. For more versions, check out:

• MobileLLM-Pro Collection
• MobileLLM-Pro-base
• MobileLLM-Pro-base Int4 CPU Quantized
• MobileLLM-Pro-base Int4 Accelerator Quantized

For an interactive demo: 🤗 Click here

Key Features

• Strong Pre-training Performance: MobileLLM-Pro base achieves impressive pre-training results, outperforming Gemma 3 1B and Llama 3.2 1B by on average 5.7% and 7.9% respectively on reasoning, knowledge, and long-context retrieval benchmarks. This performance is achieved by pre-training on less than 2T fully open-source tokens.
• 128k Context Window: The model supports up to 128k tokens, enabling long-context understanding for applications such as document summarization and information retrieval, implicitly learned from a large teacher model.
• Efficient Long-Context Inference: Interleaving local and global attention layers at a 3:1 ratio with 512 local attention, MobileLLM-Pro reduces prefill latency by 1.8x* and lowers KV cache size from 117MB to 40MB* compared to fully global attention, enabling faster and more memory-efficient inference. (*Assuming 8k context length)
• Near Lossless int4 Quantization: We provide int4 quantization-ready checkpoints for our pre-trained model with less than 1.3% quality degradation compared to floating point baselines:
  • CPU: int4 weights (group size 32), int8 dynamic activations, int8 KV cache, with only 0.4% regression.
  • Accelerators: int4 per-channel weights, with only 1.3% quality regression.
• Instruction Fine-Tuned Model: We provide a competitive instruction fine-tuned (IFT) model specializing in use-cases such as tool calling, question answering, rewriting and summarization.

MobileLLM-Pro sets a new standard for efficient, high-quality on-device language modeling. We invite the community to explore, evaluate, and build upon this model.

Model Information

Layers: 30<br> Attention Heads: 20<br> KV Heads: 4<br> Dimension: 1280<br> Hidden Dimension: 6144<br> Vocabulary Size: 202,048<br> Total Parameters: 1,084M (1.08B)

Input Modality: Text<br> Output Modality: Text<br> Languages: English<br>

Training Method: Knowledge Distillation<br> Context Length: 128k tokens<br> Teacher Model: Llama 4-Scout<br> Loss Function: KL Divergence<br> Quantization: 16-bit, 4-bit<br> Other Features: Shared Embeddings, Local-Global Attention

Model Developer: Meta Reality Labs <br> Model Release Date: October 2025 <br> License: MobileLLM-Pro is FAIR NC licensed

Results

Base Pretrained Model

FP = Full precision, bf16<br> Q-CPU = int4, group-wise quantized (for CPU)<br> Q-Acc = int4, channel-wise quantized (for Accelerators (ANE&HTP))

Instruction Tuned Model

Training Data

We constructed our datamix by selecting publicly available datasets that cover a range of domains. Using data-specific simulation runs, each dataset's contribution to the training process was carefully balanced by assigning it a specific sampling weight. These weights remained consistent throughout the base model pretraining and were informed by the extended work of Automixer and additional ablation studies. <br> The pre-training datamix primarily consists of a large educational web dataset, which makes up the vast majority of the training data. Smaller but significant portions come from coding data, mathematics, Wikipedia, scientific papers, Q&A forums, and algebraic content. In total, the datamix includes approximately 1,500 million rows and 1,640 billion tokens. <br> For our instruction fine-tuned data-mix, we focus on data diversity from existing open-source fine-tuning corpora. Specifically, we combine datasets for general instruction tuning with chat, science, safety, coding and math domains. For our final DPO phase, we rely on completely synthetic datasets.

Training Process

Pretraining

Our general pre-training process contains three distinct phases using logit-based knowledge distillation from the Llama 4-Scout model and a novel model merging paradigm:

Phase 1 (KD): Language Learning – Learn general language skills from high-quality, well balanced pre-training data <br> Phase 2 (KD): Long-context awareness – Extend the model context-length to 128k tokens using implicit positional distillation from the teacher model <br> Phase 3 (KD): Domain abilities – Acquire domain understanding through annealing of multiple models in parallel and merging the specialist models, resulting in improvements across a diverse range of domains

On top of the three pre-training phases, we add a fourth phase of Quantization-Aware Training (QAT) for our 4-bit quantized model checkpoint.

Instruction Fine-Tuning

We split the instruction fine-tuning stage into three distinct phases combining SFT and DPO methods:

Phase 1 (SFT): Learn general instruction-following with a focus on data diversity <br> Phase 2 (SFT): Domain-weight the Phase 1 data given its shortcomings (e.g. upsample code data to improve logical reasoning) <br> Phase 3 (SFT + DPO): Train and align the model for safety and self-identification

Quantization

We apply Quantization Aware Training (QAT) to our baseline and instruction fine-tuned models, yielding quantization-ready checkpoints that can either be directly converted to integer datatype (with minimal quality loss) or used for QAT on additional data. We release two quantization-ready checkpoints:

• 4-bit groupwise weight quantization with block size 32, 8-bit dynamic activations, and 8-bit kv-cache quantizations — optimized for CPU/GPU backends (xnnpack).
• 4-bit channelwise quantization without activation quantization and 8-bit kv-cache quantizations — designed for edge hardware accelerators such as Apple Neural Engine (ANE) and Qualcomm’s Hexagon Tensor Processor (HTP).

Our QAT approach incorporates long-context awareness (up to 128k tokens) and self-knowledge distillation using the full-precision teacher model. We compared the QAT-trained model to a standard round-to-nearest Post-Training Quantization (PTQ) baseline. In the groupwise pre-training setting, we observe a 34% (absolute) regression in average benchmark score when using PTQ and only a 1.5% (absolute) regression for QAT. For instruction fine-tuning, we observe less than 1% average regression using QAT.

How to use

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from huggingface_hub import login

login(token="<HF_TOKEN>")
MODEL_ID = "facebook/MobileLLM-Pro"

def generate(user_input: str, model, tokenizer) -> str:
    user_input = [{"role": "user", "content": user_input}]
    inputs = tokenizer.apply_chat_template(
      user_input, return_tensors="pt", add_generation_prompt=True
    ).to(model.device)
    outputs = model.generate(inputs, max_new_tokens=128)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

def main():
    tokenizer = AutoTokenizer.from_pretrained(
        MODEL_ID, trust_remote_code=True
    )
    model = AutoModelForCausalLM.from_pretrained(
        MODEL_ID, trust_remote_code=True
    )
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    model.eval()

    prompt = "Why are open-source on-device language models great?"
    result = generate(prompt, model, tokenizer)
    print(result)

if __name__ == "__main__":
    main()

Latency benchmarking

Latency benchmarking was done on a Samsung Galaxy S25 CPU and Samsung Galaxy S24 Hexagon Tensor Processor (HTP). Models were exported to ExecuTorch with XNNPACK backend (for CPU) and HTP backend (for accelerator). The model size of the CPU model with 4-bit groupwise quantization is 590MB. The CPU and HTP prefill latency for different input prompt lengths of 2k, 4k and 8k along with decode speed for generating 1k tokens is shown in the following table.

To validate the benefit of interleaved local-global attention (LGA), we benchmark models across different prompt lengths and measure the speed-up in prefill & decode relative to using global attention at every layer:

Citation

@misc{mobilellm_pro,<br> title={MobileLLM-Pro Model Card},<br> author={Patrick Huber*, Ernie Chang*, Wei Wen*, Igor Fedorov*, Tarek Elgamal, Hanxian Huang, Naveen Suda, Chinnadhurai Sankar, Vish Vogeti, Yanghan Wang, Alex Gladkov, Kai Sheng Tai, Abdelrahman Elogeel, Tarek Hefny, Vikas Chandra, Ahmed Aly, Anuj Kumar, Raghuraman Krishnamoorthi**, Adithya Sagar**}, <br> year={2025},<br> month={October},<br> url = {https://huggingface.co/facebook/MobileLLM-Pro}}

Contact

Patrick Huber, Meta Inc, Reality Labs ([email protected])<br> Ernie Chang, Meta Inc, Reality Labs ([email protected])<br> Wei Wen, Meta Inc, Reality Labs ([email protected])<br> Igor Fedorov, Meta Inc, Reality Labs ([email protected])<br> Raghuraman Krishnamoorthi, Meta Inc Reality Labs ([email protected])<br> Adithya Sagar, Meta Inc, Reality Labs ([email protected])

Acknowledgements

We want to thank the team involved in this project, especially: Kimish Patel, Andrew Or, Min Guo, Shen Xu, Brian Moran, Maho Takahashi, Claire Lesage, Rylan Conway, Karan Chadha, Matthew Grange, Tomasz Wołcyrz, Shiv Desai, Amarlin Anand, Joele Sires, Robert Carrillo, Francisc Bungiu, Jayden Yu, AJ Brush, Yang Li, Samuel Selvan, Anand Sharma, Peng Shan, Anand Dass, Abhishek Sharma

License

MobileLLM-Pro is distributed under the FAIR NC license