ticket-classifier-lora

Model Card: Ticket Classifier (Llama-3.2-1B + LoRA)

Model Description

A LoRA adapter fine-tuned on top of meta-llama/Llama-3.2-1B for support ticket classification.

Intended Use

Classify customer support tickets into one of five categories:

• billing — payment, invoice, charge, refund queries
• technical — bugs, errors, product not working
• account — login, password, profile, access issues
• shipping — delivery, tracking, lost package queries
• general — feedback, feature requests, and inquiries that don't fit a specific support category

Out-of-scope use: This model is not intended for legal- or compliance-sensitive ticket routing without human review, and has not been evaluated on real customer data or non-English tickets.

Training Data

600 synthetic support tickets generated by claude-sonnet-4-6 — 120 per category. Tickets vary in length (1–5 sentences) and tone (frustrated, polite, confused, urgent). No real customer data was used.

Generation used explicit per-category instructions (rather than a single generic prompt) so that each category — including general — has a clear positive definition. An earlier version of this dataset used an undefined general category, which caused the generation model to fill it with tickets indistinguishable from the other four categories; this was corrected before the final training run below.

Known limitation: Class balance (120/category) is artificial. Real-world ticket distributions are rarely this even, so reported metrics may not reflect performance under real class imbalance.

Training Details

Evaluation

Evaluated on a held-out 20% split (120 examples) of the synthetic dataset.

Final validation metrics: accuracy 0.951, F1 (macro) 0.953, loss 0.248.

Confusion matrix summary: billing, shipping, and general were classified with zero errors. The remaining errors (5 of 120 tickets) were concentrated between technical and account, likely reflecting genuine overlap (e.g. login issues that are also technical errors) rather than a labeling artifact.

An earlier training run on an unrefined dataset (undefined general category) scored 71.7% accuracy / 0.704 F1 (macro) on the same model architecture and hyperparameters, with most errors concentrated in general misclassifications. This gap was traced to a data generation issue rather than a modeling issue — see Training Data above.

Limitations

• Trained and evaluated entirely on synthetic data — has not been validated against real customer support tickets, which may differ in style, length, or ambiguity
• Five fixed categories — tickets spanning multiple categories default to the closest match
• Small residual confusion between technical and account categories
• English only
• Not evaluated for adversarial inputs or prompt injection
• Artificially balanced training classes (see Training Data)

License

This adapter is released under Apache 2.0. The base model, meta-llama/Llama-3.2-1B, is subject to Meta's Llama 3.2 Community License, which includes usage restrictions (e.g. on very large-scale commercial deployments and certain use cases). Review the Llama 3.2 license before deploying this adapter.

How to Use

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "meta-llama/Llama-3.2-1B"
ADAPTER = "sheethal00/ticket-classifier-lora"

id2label = {0: "billing", 1: "technical", 2: "account", 3: "shipping", 4: "general"}
label2id = {v: k for k, v in id2label.items()}

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForSequenceClassification.from_pretrained(
    BASE_MODEL,
    num_labels=5,
    id2label=id2label,
    label2id=label2id,
    torch_dtype=torch.float16,  # use torch.float32 if running on CPU
)
base_model.config.pad_token_id = tokenizer.pad_token_id

model = PeftModel.from_pretrained(base_model, ADAPTER)
model.eval()

# Inference
text = "I was charged twice for my subscription this month, can you refund the extra charge?"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_id = logits.argmax(dim=-1).item()
print(model.config.id2label[predicted_id])  # -> "billing"

Note: This adapter was trained with 4-bit quantization (QLoRA). For inference, full precision (fp16/fp32) works fine and is simpler to set up; if you want to match the training setup exactly, load base_model with a BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16) (requires a CUDA GPU — 4-bit quantization is not supported on CPU).