support_ticket_llm

Fine-Tuning Phi-3-mini for Support Ticket Extraction

Fine-tunes an LLM with Unsloth to turn free-text customer complaints/emails into structured JSON: product, category, urgency, sentiment.

Model

• Base model: unsloth/Phi-3-mini-4k-instruct-bnb-4bit
• Microsoft's Phi-3-mini (3.8B params, instruction-tuned), re-packaged by Unsloth in 4-bit quantized form for lightweight fine-tuning (e.g. free Colab T4 GPU).

Files

Workflow (Colab)

1. Runtime → Change runtime type → Python 3 + T4 GPU
2. !pip install unsloth
3. Load model with FastLanguageModel.from_pretrained(...)
4. Format dataset into chat-template text (tokenizer.apply_chat_template)
5. Apply LoRA (r=64, attention + MLP layers)
6. Train with SFTTrainer (max_steps=60, batch size 2, grad accumulation 4)
7. Quick test with model.generate()
8. Export to GGUF: model.save_pretrained_gguf(..., quantization_method="q4_k_m")

Running Locally (Ollama)

brew install ollama
ollama create support-ticket-phi3 -f Modelfile
ollama run support-ticket-phi3

Test with multiple cases:

while IFS= read -r line; do
  echo "--- Input: $line ---"
  ollama run support-ticket-phi3 "$line"
  echo ""
done < inferece_test.txt

Result test case:

ollama run support-ticket-phi3                
>>> I ordered a MALM bed frame last week and one of the side panels arrived with a big scratch. Not a huge deal but I'd like a replacement panel sent over.
{"category": "product defect", "product": "MALM bed frame", "sentiment": "neutral", "urgency": "low"}

ollama run support-ticket-phi3                
>>> Loved how fast your support team replied when I asked about my Adidas Ultraboost order, really appreciated the quick and friendly help.
{"category": "feedback", "product": "Adidas Ultraboost", "sentiment": "positive", "urgency": "low"}

ollama run support-ticket-phi3      
>>> The Samsung Galaxy Buds I bought stopped connecting to my phone after only a few days. Pretty frustrating since I use them daily for calls.
{"category": "product defect", "product": "Samsung Galaxy Buds", "sentiment": "negative", "urgency": "medium"}

Known Limitations

• Tiny dataset (12 examples) — fine for demo purposes, but not enough for real generalization. Scale up to hundreds of varied examples for production.
• Model loses general-purpose ability after fine-tuning — since every training example follows "free text → JSON", the model always replies in JSON, even for unrelated questions (e.g. "who is Isaac Newton?" still returns a JSON blob). This isn't a bug — it's expected from small, single-pattern fine-tuning. Fix by mixing in:
  • Explicit instructions per prompt ("Extract this ticket into JSON: ...")
  • Regular conversational examples alongside extraction examples
  • A consistent system prompt at both training and inference time
• Default temperature = 1.5 in the auto-generated Modelfile is too high for structured extraction — lower it to 0.1–0.3 for consistent JSON.

Compatibility Notes

• Training requires an NVIDIA GPU (CUDA) — won't run on Mac/Apple Silicon, since unsloth and bitsandbytes depend on Triton/CUDA. Use Colab's free T4 GPU or another CUDA cloud.
• Inference on the final .gguf file is lightweight and runs fine on CPU/Apple Silicon via Ollama or llama.cpp — no GPU needed.