lab21-qwen35-9b-triage-lora

Lab21 Qwen3.5-9B Vietnamese Triage LoRA

LoRA adapter fine-tuned from Qwen/Qwen3.5-9B for the VinUni AICB Day 21 Track 3 fine-tuning lab.

Task

The model classifies Vietnamese customer-support tickets into JSON with four fields:

• intent
• urgency
• product
• sentiment

Base Model

Qwen/Qwen3.5-9B

LoRA Configuration

• Rank: 16
• LoRA alpha: 32
• Placement: text linear layers
• Trainable parameters: 43,278,336
• Precision: BF16
• Learning rate: 1e-4
• Training steps: 30
• Mask mode: assistant-only

Evaluation

The LoRA adapter substantially improves the target task from 0.815 to 0.985.

However, the regression score decreases from 0.7422 to 0.1333.

Therefore, the regression gate verdict is:

FAILED

This adapter is strongly specialized for the customer-support triage task and should not be treated as a general-purpose replacement for the base model.

Limitations

The fine-tuned model shows domain over-specialization and catastrophic forgetting on general questions.

For example, general-knowledge questions may be incorrectly converted into customer-support JSON structures instead of being answered normally.

A possible next experiment is to mix approximately 1–5% general replay data into the training dataset and rerun the same regression gate.

Files

This repository contains the PEFT LoRA adapter and supporting tokenizer/template files:

• adapter_model.safetensors
• adapter_config.json
• tokenizer.json
• tokenizer_config.json
• chat_template.jinja

The original Qwen/Qwen3.5-9B base weights are not included.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "Qwen/Qwen3.5-9B"
adapter_id = "backache/lab21-qwen35-9b-triage-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype="auto",
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    adapter_id,
)

model.eval()

Training Summary

The main LoRA run used:

• 250 Vietnamese customer-support samples
• Train/validation split: 225 / 25
• max_length=2048
• assistant-only loss masking
• 2 epochs
• 30 optimizer steps
• Peak VRAM: 23.26 GB
• Final training loss: 0.2797

Training was performed on an NVIDIA L40 GPU.

Repository

GitHub:

https://github.com/vinhphamv2303-arch/Day21-Track3-Finetuning-Lab

Notes

The adapter achieved strong task-specific performance but failed the general-capability regression gate. The evaluation results are intentionally reported as-is rather than modifying the regression threshold or evaluation set.