AfriqueQwen3.5-4B-50Langs

AfriqueQwen3.5-4B-50Langs

Model Overview

AfriqueQwen3.5-4B-50Langs is part of the AfriqueLLM suite, a collection of open language models adapted to 50 African languages through continued pre-training (CPT) on ~35.5B tokens. This model is based on Qwen/Qwen3.5-4B-Base and has been specifically adapted for improved performance on African languages while maintaining strong capabilities in high-resource languages.

This variant extends AfriqueQwen3.5-4B-ExtendedCM with approximately 1.5B additional tokens from remaining African language data, bringing the supported African language coverage to 50 languages.

Key Features

• Type: Causal Language Model (Base/Pre-trained)
• Base Model: Qwen 3.5 4B
• Parameters: 4B
• Context Length: 32,768 tokens (native)
• Training Tokens: ~35.5B tokens of carefully curated multilingual data

Supported Languages

AfriqueQwen3.5-4B-50Langs has been adapted for the following 50 African languages:

High-resource languages used for catastrophic forgetting mitigation: English, French, Portuguese, Arabic

Training Data

Our training corpus combines multiple high-quality sources:

• African Monolingual Data (~22.8B tokens): FineWeb2, WURA, and MADLAD-400
• Code (~1B tokens): CornStack-Python for reasoning capabilities
• Mathematics (~1B tokens): FineMath-4+ for mathematical understanding
• Synthetic Data (~324M tokens): GPT-4.1 translated domain-specific content across 10 domains
• Additional Language Expansion (~1.5B tokens beyond AfriqueQwen3.5-4B-ExtendedCM): remaining African language data, upsampled 5x for broader language coverage.

We use UniMax sampling to create a balanced distribution, capping high-resource languages at approximately 1B tokens and upsampling lower-resource languages for up to five epochs.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "McGill-NLP/AfriqueQwen3.5-4B-50Langs"

# Load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Prepare the model input
prompt = "Bawo ni o ṣe n ṣe?"  # Yoruba: "How are you doing?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text
generated_ids = model.generate(
    **inputs,
    max_new_tokens=100,
)
output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(output)

Deployment

For deployment, you can use vllm or sglang to create an OpenAI-compatible API endpoint:

vLLM:

vllm serve McGill-NLP/AfriqueQwen3.5-4B-50Langs

SGLang:

python -m sglang.launch_server --model-path McGill-NLP/AfriqueQwen3.5-4B-50Langs

Training Details

Hyperparameters

• Learning Rate: 5e-5 (with warmup and cosine decay)
• Context Length: 16,384 tokens
• Optimizer: AdamW
• Precision: BF16 mixed precision

Infrastructure

Training was conducted using the LLaMA-Factory framework on up to 64 NVIDIA H100 GPUs with:

• DeepSpeed ZeRO-1/ZeRO-2
• Flash Attention 3
• Sequence packing
• Liger Kernel optimizations

Evaluation

All AfriqueLLM models are evaluated on multiple multilingual benchmarks. FLORES is reported only in the English-to-target direction (eng->xxx):

Additional-Language Evaluation

This table averages only evaluated African languages outside the first 20-language CPT set: Ewe, Lingala, Ganda, Twi, and Wolof. Benchmark cells average the available languages for that benchmark; FLORES is English-to-target only (eng->xxx).

Model Variants

• AfriqueQwen-14B - Qwen-based 14B model (flagship)
• AfriqueQwen-8B - Qwen-based 8B model
• AfriqueQwen-4B - Qwen-based 4B model
• AfriqueQwen3.5-4B - Qwen 3.5-based 4B model
• AfriqueQwen3.5-4B-ExtendedCM - Qwen 3.5-based 4B model with extended continued pre-training
• AfriqueGemma-4B - Gemma-based 4B model
• AfriqueGemma-12B - Gemma-based 12B model
• AfriqueLlama-8B - Llama-based 8B model

Citation

If you find our work helpful, please cite:

@misc{yu2026afriquellmdatamixingmodel,
      title={AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages}, 
      author={Hao Yu and Tianyi Xu and Michael A. Hedderich and Wassim Hamidouche and Syed Waqas Zamir and David Ifeoluwa Adelani},
      year={2026},
      eprint={2601.06395},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.06395}, 
}

License

This model is released under the CC BY 4.0 License. Please review the license terms before use.

Acknowledgments

We thank the creators of the base models, datasets and compute resources that made this work possible, including Mila, Compute Canada, Microsoft, the FineWeb team, WURA, MADLAD-400 and etc..