qwen2.5-0.5b-airline-triage-lora

Qwen2.5-0.5B Airline Ticket Triage (LoRA)

LoRA adapters that turn Qwen2.5-0.5B-Instruct into a classifier for airline customer-support messages. Given a tweet, the model returns JSON with an intent (10 classes), an urgency level (3 levels, time-based criterion) and an abusive flag.

Trained on 5,130 tweets from the Customer Support on Twitter corpus, labeled by DeepSeek V4 Flash and audited against a 300-message human-reviewed gold set (labeler–human agreement: intent 97.7%, urgency 92.3%).

Full pipeline (labeling, audit, training, FastAPI serving, AWS deployment): github.com/eorgantzoglou/Smart-Support-Ticket-Triage

Results

Evaluated on the 300-message gold set against human labels — the gold set was fully excluded from training:

The zero-shot base model classifies essentially everything as general_question — the classification ability comes from these adapters. Errors concentrate in rare classes (special_assistance: F1 0.29 from only 121 training examples); frequent classes do well (praise 0.91, delay 0.83, lost luggage 0.80).

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "eorgantzoglou/qwen2.5-0.5b-airline-triage-lora"

SYSTEM_PROMPT = (
    "You are a triage assistant for an airline's customer support. "
    "Classify the customer tweet. Respond with json only, in exactly this format: "
    '{"intent": "...", "urgency": "...", "abusive": true/false}. '
    "intent must be one of: delay_disruption, checkin_boarding_issue, "
    "flight_cancellation_rebooking, lost_luggage, special_assistance, "
    "general_complaint, general_question, praise_feedback, spam_irrelevant, "
    "other_unclear. urgency must be one of: high, medium, low."
)

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float32), ADAPTER
).merge_and_unload()

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "my bags are lost and nobody at the desk is helping"},
]
inputs = tokenizer(
    tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True),
    return_tensors="pt",
)
out = model.generate(**inputs, max_new_tokens=60, do_sample=False,
                     pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# {"intent": "lost_luggage", "urgency": "high", "abusive": false}

The system prompt must match the one used in training (above) — the model was fine-tuned with it and expects it verbatim.

Training

• LoRA: r=16, α=32, dropout 0.05, adapters on all linear layers (q/k/v/o/gate/up/down) — ~35 MB of weights
• 2 epochs, lr 2e-4, fp16, on a free Colab T4
• Chat-format dataset (system/user/assistant), 5,130 train / 570 val

Limitations

• The abusive flag is unreliable: only 10 of 6,000 training examples were abusive. In production this signal should come from a separate moderation layer, not this classifier.
• Rare intents (special_assistance, spam_irrelevant) have low F1 due to class imbalance.
• English only; trained on airline-domain tweets — expect degradation on other support domains.