improved-aya-expanse-8b-cpt-tunisian

Improved Aya Expanse 8B CPT Tunisian

This is a LoRA adapter trained on the Tunisian Arabic dialect using the Aya Expanse 8B base model through Continued Pre-Training (CPT).

Model Details

Model Description

• Base Model: CohereLabs/aya-expanse-8b (8B multilingual)
• Training Method: QLoRA (4-bit quantization + LoRA adapter)
• Language: Tunisian Arabic (Darija)
• Dataset: Syrinesmati/tunisian-dialect-corpus (1.18M samples → 82.6k packed sequences)
• Training Steps: 5,113 (1 epoch)
• Final Eval Loss: 2.089 (↓ 16% improvement from base)
• License: Apache 2.0

Model Sources

• Base Model Repository: CohereLabs/aya-expanse-8b
• Dataset Repository: Syrinesmati/tunisian-dialect-corpus

Uses

Direct Use - Quick Start

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "CohereLabs/aya-expanse-8b",
    device_map="auto",
    trust_remote_code=True
)

# Load Tunisian adapter
model = PeftModel.from_pretrained(
    base_model,
    "alabenayed/improved-aya-expanse-8b-cpt-tunisian",
    device_map="auto"
)
model.eval()

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "alabenayed/improved-aya-expanse-8b-cpt-tunisian",
    trust_remote_code=True
)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Generate with chat template
def generate_prompt(prompt: str, max_new_tokens: int = 160):
    messages = [{"role": "user", "content": prompt}]
    inputs = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    )
    
    device = next(model.parameters()).device
    inputs = {k: v.to(device) for k, v in inputs.items()}
    
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        temperature=0.7,
        do_sample=True,
    )
    
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Try with Tunisian prompts
prompts = [
    "عسلامة تحكي تونسي ؟",  # How are you speaking Tunisian?
    "اليوم الطقس مزيان شنوة نجم نعمل؟",  # Today's weather is nice, what can we do?
    "اش نجم نقول لصاحبي باش نهنيه بعيد ميلادو؟",  # What can I say to my friend on his birthday?
]

for prompt in prompts:
    output = generate_prompt(prompt)
    print(f"📝 {prompt}\n💬 {output}\n")

Use Cases

✅ Tunisian Arabic text generation
✅ Conversational AI in Tunisian dialect
✅ Fine-tuning base for downstream tasks
✅ Multilingual models with Tunisian support

Downstream Use

Use this adapter with other Tunisian datasets or for domain-specific fine-tuning tasks.

Training Details

Dataset

• Source: Syrinesmati/tunisian-dialect-corpus
• Raw samples: 1,180,174
• After packing: 82,625 sequences (1024 tokens each)
• Train/Eval split: ~98.8% / 1.2%

Training Procedure

Configuration

• Quantization: 4-bit (NF4 + double quantization)
• LoRA Settings:
  • r=16 (rank)
  • α=32 (scaling)
  • dropout=0.05
• Optimizer: AdamW
• Learning Rate: 2e-4 (cosine with warmup)
• Warmup Steps: 153
• Batch Size: 1 (with gradient accumulation)
• Total Steps: 5,113
• Epochs: 1

Performance

• Training Duration: ~72 hours on single NVIDIA GPU
• Final Eval Loss: 2.089
• Improvement: ~16% loss reduction from base model

Loss Progression

Evaluation Results

Final Metrics

• Eval Loss: 2.089
• Perplexity: 8.08
• Improvement over base: 16% loss reduction

Limitations

⚠️ Best performance on Tunisian Arabic text
⚠️ May require additional fine-tuning for domain-specific tasks
⚠️ Inherits limitations from base Aya Expanse 8B model

Citation

@misc{tunisian-llm-2026,
    author = {MohamedAla-Benayed},
    title = {Improved Aya Expanse 8B CPT Tunisian},
    year = {2026},
    publisher = {Hugging Face},
    howpublished = {\url{https://huggingface.co/alabenayed/improved-aya-expanse-8b-cpt-tunisian}}
}

Acknowledgments

• Base Model: Cohere - Aya Expanse 8B
• Dataset: Syrinesmati - Tunisian Dialect Corpus
• Framework: Hugging Face Transformers & PEFT

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]

Framework versions

• PEFT 0.19.1