AraCompose-Qwen3-8B-adapters

AraCompose: Zero-Shot Arabic Task Transfer via Cross-Lingual Adapter Composition

Paper: "AraCompose: Zero-Shot Arabic Task Transfer via Cross-Lingual Adapter Composition with FLARE and TIES Merging" Authors: Mark Kashirskiy, Artiom Lipinski, Ilya Makarov Base model: Qwen/Qwen3-8B

Overview

AraCompose decomposes Arabic task performance into English task adapters + Arabic language adapter, then composes them via AdaMergeX. The result is Arabic task performance with zero Arabic task labels.

Method

1. Train English task adapters (QA, Math, Instruction, Safety) on SQuAD/GSM8K/Alpaca/SafetyBench
2. Train Arabic language adapter (CLM on Arabic Wikipedia)
3. Apply AdaMergeX formula: A_composed = A_ar + t * (A_en_task - A_en_wiki)
4. Optionally compose multiple task adapters via TIES merging

Adapter Variants

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = "Qwen/Qwen3-8B"
adapter_path = "mariklolik228/AraCompose-Qwen3-8B-adapters/full_adapters/adamergex_qa"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    load_in_4bit=True,
    device_map={"": 0}
)
model = PeftModel.from_pretrained(model, adapter_path)

# Generate Arabic answer to an English QA question (zero-shot transfer)
prompt = "أجب على السؤال التالي: ما هي عاصمة مصر؟"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Citation

@article{kashirskiy2026aracompose,
  title={AraCompose: Zero-Shot Arabic Task Transfer via Cross-Lingual Adapter Composition},
  author={Kashirskiy, Mark and Lipinski, Artiom and Makarov, Ilya},
  year={2026}
}