llama3.1-8b-ar-ticket

Llama 3.1 8B — Arabic Maintenance Ticket Cleaner & Translator

A LoRA adapter fine-tuned on top of meta-llama/Llama-3.1-8B-Instruct (trained against the ungated mirror NousResearch/Meta-Llama-3.1-8B-Instruct) that converts raw, informal Arabic workplace maintenance tickets into clean, structured English JSON.

Model Details

Model Description

Employees report workplace maintenance problems in Arabic, often with typos, informal phrasing, repeated information across title/description, and multiple unrelated issues bundled into a single ticket. This model is a preprocessing step that runs before an English ticket classifier: it takes the raw Arabic ticket and produces a cleaned-up, translated, well-structured English version, plus a short explanation of what was changed.

• Developed by: [your name / org]
• Model type: Causal language model, LoRA adapter (not a merged/standalone model)
• Language(s): Input: Arabic (with code-switched English fragments). Output: English.
• License: Llama 3.1 Community License (inherited from the base model)
• Finetuned from model: meta-llama/Llama-3.1-8B-Instruct / NousResearch/Meta-Llama-3.1-8B-Instruct

Model Sources

• Base model: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
• Ungated mirror used for training: https://huggingface.co/NousResearch/Meta-Llama-3.1-8B-Instruct

Uses

Direct Use

Given a raw Arabic maintenance ticket (title + description), the model outputs a JSON object with a cleaned English title, description, and a reasoning field explaining the edits made. Intended to run as an automated preprocessing step immediately upstream of an English-language ticket classifier — not intended as a general-purpose Arabic-English translator or general-purpose chat assistant.

Out-of-Scope Use

• General machine translation outside the maintenance-ticket domain.
• Domains/languages other than Arabic-to-English workplace maintenance tickets — the model has not been evaluated on other text types and may not generalize.
• Any use where an incorrect or hallucinated translation could cause real-world harm without human review (e.g. safety-critical maintenance dispatch) — outputs should be spot-checked, especially early in deployment.

How to Get Started with the Model

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL_ID = "NousResearch/Meta-Llama-3.1-8B-Instruct"
ADAPTER_DIR = "./llama31-8b-ar-ticket-cleaner-final"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL_ID, quantization_config=bnb_config, device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER_DIR)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_DIR)  # loads the training chat template
tokenizer.pad_token = tokenizer.pad_token or tokenizer.eos_token

SYSTEM_PROMPT = """You are a workplace maintenance ticket cleaner and translator.
You receive a raw Arabic maintenance ticket (title and description) and must output ONLY a JSON object with three fields: "title", "description", "reasoning".

Rules:
- Fix spelling/typos without changing the meaning.
- If the title and description repeat the same issue, merge them.
- If a ticket has multiple distinct issues, split the description into bullet points (one per issue).
- Translate place names to their standard English form.
- Normalize all variant spellings of store signage (جارمة، الجارمة، ارمات، الارمات، قارمة، القارمه، قارمه، قارمت، القارمة، للقارمه، بقارما) to the same English term.
- Translate everything to clear, professional English.
- "reasoning" is a short note explaining what was fixed and how it was translated.
Output valid JSON only, no extra text."""

def clean_ticket(source_title, source_description):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": json.dumps(
            {"title": source_title, "description": source_description}, ensure_ascii=False)},
    ]
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=300, do_sample=False)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print(clean_ticket("ماكينة الباريستا", "مشكلة بجروبين الاول ما بشبك و الثاني بنزل ماء من الاطراف"))

Note: this repository contains a LoRA adapter, not a standalone model. You must load the base model (NousResearch/Meta-Llama-3.1-8B-Instruct or, once access is approved, meta-llama/Llama-3.1-8B-Instruct) and attach this adapter with peft.PeftModel.from_pretrained, as shown above. Load the tokenizer from the adapter directory, not the base model, since the saved chat_template.jinja contains generation-span markers required for the model to have been trained correctly.

Training Details

Training Data

A dataset of Arabic maintenance tickets paired with cleaned English versions and a reasoning explanation. Each row: case_id, source_title, source_description (Arabic input), target_title, target_description, reasoning (English target, combined into a single JSON object as the training label).

Cleaning rules encoded in training data and the system prompt:

• Fix spelling/typos without changing meaning.
• Remove repetition between title and description.
• Split multiple distinct issues into bullet points.
• Translate place names to standard English form.
• Normalize all variant spellings of store signage — جارمة، الجارمة، ارمات، الارمات، قارمة، القارمه، قارمه، قارمت، القارمة، للقارمه، بقارما — to the same English term.

Training Procedure

QLoRA fine-tuning: base model loaded in 4-bit (NF4, double quantization, bf16 compute dtype), LoRA adapter trained on top.

Preprocessing

Each example formatted as a system / user / assistant message list (not a pre-flattened string), so that trl's assistant_only_loss masking correctly restricts the training loss to the assistant's JSON output only. The base model's default chat template was patched with explicit {% generation %} / {% endgeneration %} markers to support this masking.

Training Hyperparameters

• LoRA: r=16, alpha=32, dropout=0.05, targeting q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Quantization: 4-bit NF4, double quantization, bf16 compute dtype
• Batch size: 4 per device, gradient accumulation 4 (effective batch size 16)
• Epochs: 3
• Learning rate: 2e-4, cosine schedule, 3% warmup
• Sequence length: 1024 tokens
• Precision: bf16
• Loss masking: assistant-turn-only (assistant_only_loss=True)

(Update the above with the actual final values used, if they were changed from these defaults during experimentation.)

Evaluation

Testing Data

Held-out validation split of the same dataset, including a manually curated subset of case IDs chosen to stress-test known tricky patterns: signage-word spelling variants (قائمة/القارمة/القارمه/بقارما), multi-issue tickets (numbered lists, "+"-joined issues), and single vs. merged-issue tickets.

Metrics

• JSON validity rate: percentage of outputs that parse as valid JSON with exactly the three expected keys (title, description, reasoning).
• Semantic similarity: cosine similarity (via sentence-transformers, all-mpnet-base-v2 embeddings) between model output and ground truth, computed separately for title, description, and combined title+description, averaged across the validation set.

Results

(Fill in with the actual numbers from your evaluation run.)

Known Limitations

• On manual review, the model sometimes preserves surface-level word overlap with the source while inverting or softening the actual meaning (e.g. describing a broken component as functioning, or describing an installation/wiring task as a "malfunction") — semantic similarity metrics can under-penalize these cases since vocabulary overlap remains high even when meaning has flipped. Manual spot-checking on meaning-critical fields is recommended in addition to automated similarity metrics.
• Not evaluated outside the maintenance-ticket domain; unlikely to generalize to general Arabic-English translation tasks.
• Trained on a specific set of store-signage spelling variants; novel/unseen misspellings of the same word may not be normalized correctly.

Environmental Impact

(Optional — fill in if you want to report compute/carbon estimates, e.g. via the ML CO2 Impact calculator.)

• Hardware Type: TBD
• Hours used: TBD
• Cloud Provider: TBD
• Compute Region: TBD

Technical Specifications

Model Architecture and Objective

Standard Llama 3.1 8B decoder-only transformer architecture (unmodified), fine-tuned via LoRA adapters injected into attention (q_proj, k_proj, v_proj, o_proj) and MLP (gate_proj, up_proj, down_proj) projection layers, with the base weights frozen. Training objective: standard next-token cross-entropy loss, masked to the assistant's JSON output span only.

Compute Infrastructure

(Fill in: GPU type/count, cloud provider or local, training time.)

Framework Versions

• transformers
• peft
• trl 1.8.0
• bitsandbytes 0.49.2
• PEFT (LoRA)